{"paragraphs":[{"text":"%md\n### Note\n\nPlease view the [README](https://github.com/deeplearning4j/deeplearning4j/tree/master/dl4j-examples/tutorials/README.md) to learn about installing, setting up dependencies, and importing notebooks in Zeppelin","user":"anonymous","dateUpdated":"2017-11-24T06:12:25+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Note</h3>\n<p>Please view the <a href=\"https://github.com/deeplearning4j/deeplearning4j/tree/master/dl4j-examples/tutorials/README.md\">README</a> to learn about installing, setting up dependencies, and importing notebooks in Zeppelin</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1511371747465_366506506","id":"20171020-070156_1850232313","dateCreated":"2017-11-22T17:29:07+0000","dateStarted":"2017-11-24T06:12:25+0000","dateFinished":"2017-11-24T06:12:25+0000","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:6536"},{"text":"%md\n\n### Background\n\n#### Cloud Cover\nBy definition, _cloud cover_ is the portion of the _sky_ covered by _clouds_ when viewed from a particular location (such as a weather station). It is usually measured in a unit called [__Okta__](https://en.wikipedia.org/wiki/Okta). Cloud cover is also helpful in determining sunshine duration as it is inversly related to cloud cover.\n\n--- \n\n#### Goals\n- Determining cloud cover in DL4J","user":"anonymous","dateUpdated":"2017-11-24T06:12:25+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Background</h3>\n<h4>Cloud Cover</h4>\n<p>By definition, <em>cloud cover</em> is the portion of the <em>sky</em> covered by <em>clouds</em> when viewed from a particular location (such as a weather station). It is usually measured in a unit called <a href=\"https://en.wikipedia.org/wiki/Okta\"><strong>Okta</strong></a>. Cloud cover is also helpful in determining sunshine duration as it is inversly related to cloud cover.</p>\n<hr/>\n<h4>Goals</h4>\n<ul>\n  <li>Determining cloud cover in DL4J</li>\n</ul>\n</div>"}]},"apps":[],"jobName":"paragraph_1511371747467_-202076509","id":"20171020-070208_2069142559","dateCreated":"2017-11-22T17:29:07+0000","dateStarted":"2017-11-24T06:12:25+0000","dateFinished":"2017-11-24T06:12:25+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:6537"},{"text":"%md\n\n## 1. Determining cloud cover in DL4J\n\nWe're going to translate [this code](https://github.com/bpark738/Cloud/tree/master/src/main/java/stat215) into scala and visualize it in zeppelin's notebook format. The code models cloud detection in polar regions based on __radiances__ recorded automatically by the ___MISR sensor___ aboard the NASA satellite, ___Terra___.\n","user":"anonymous","dateUpdated":"2017-11-24T06:17:54+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h2>1. Determining cloud cover in DL4J</h2>\n<p>We&rsquo;re going to translate <a href=\"https://github.com/bpark738/Cloud/tree/master/src/main/java/stat215\">this code</a> into scala and visualize it in zeppelin&rsquo;s notebook format. The code models cloud detection in polar regions based on <strong>radiances</strong> recorded automatically by the <strong><em>MISR sensor</em></strong> aboard the NASA satellite, <strong><em>Terra</em></strong>.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1511371747468_1317578123","id":"20171116-134509_791025875","dateCreated":"2017-11-22T17:29:07+0000","dateStarted":"2017-11-24T06:17:54+0000","dateFinished":"2017-11-24T06:17:54+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:6538"},{"text":"%md\n## 2. CNN in DL4J\n- #### Data\n    You can view or download the data from [here](https://github.com/bpark738/Cloud/tree/master/src/main/resources). The dataset has 9 train/test splits for our convenience.\n\n- #### Data features\n    - 3 satellite [images](https://github.com/bpark738/Cloud/tree/master/images).\n    - __Expert labels__ used for model training for each point in image (see the images in the table below).\n    - __NDAI, SD, CORR__, based on [subject knowledge](https://github.com/bpark738/Cloud/blob/master/yu2008.pdf).\n    - __DF, CF, BF, AF, AN__ [(Radiance angles)](http://www-misr.jpl.nasa.gov/).\n\n- #### Images\n\n|Image 1|Image 2|Image 3|\n|---|---|---|\n|![Image 1](https://raw.githubusercontent.com/bpark738/Cloud/master/images/image1.png)|![Image 2](https://raw.githubusercontent.com/bpark738/Cloud/master/images/image2.png)|![Image 3](https://raw.githubusercontent.com/bpark738/Cloud/master/images/image3.png)|\n\n|Figures|\n|---|\n|In the above table, the figure shows the regions highlighted though colored labels - ___ice___ as __'red'__, ___clouds___ as __'blue'__ and ___unknowns___ as __'green'__|\n","user":"anonymous","dateUpdated":"2017-11-24T06:24:55+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{"0":{"graph":{"mode":"table","height":386.188,"optionOpen":false}}},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h2>2. CNN in DL4J</h2>\n<ul>\n  <li>\n  <p>\n  <h4>Data</h4>You can view or download the data from <a href=\"https://github.com/bpark738/Cloud/tree/master/src/main/resources\">here</a>. The dataset has 9 train/test splits for our convenience.</p></li>\n  <li>\n    <h4>Data features</h4>\n    <ul>\n      <li>3 satellite <a href=\"https://github.com/bpark738/Cloud/tree/master/images\">images</a>.</li>\n      <li><strong>Expert labels</strong> used for model training for each point in image (see the images in the table below).</li>\n      <li><strong>NDAI, SD, CORR</strong>, based on <a href=\"https://github.com/bpark738/Cloud/blob/master/yu2008.pdf\">subject knowledge</a>.</li>\n      <li><strong>DF, CF, BF, AF, AN</strong> <a href=\"http://www-misr.jpl.nasa.gov/\">(Radiance angles)</a>.</li>\n    </ul>\n  </li>\n  <li>\n  <h4>Images</h4></li>\n</ul>\n<table>\n  <thead>\n    <tr>\n      <th>Image 1</th>\n      <th>Image 2</th>\n      <th>Image 3</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <td><img src=\"https://raw.githubusercontent.com/bpark738/Cloud/master/images/image1.png\" alt=\"Image 1\" /></td>\n      <td><img src=\"https://raw.githubusercontent.com/bpark738/Cloud/master/images/image2.png\" alt=\"Image 2\" /></td>\n      <td><img src=\"https://raw.githubusercontent.com/bpark738/Cloud/master/images/image3.png\" alt=\"Image 3\" /></td>\n    </tr>\n  </tbody>\n</table>\n<table>\n  <thead>\n    <tr>\n      <th>Figures</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <td>In the above table, the figure shows the regions highlighted though colored labels - <strong><em>ice</em></strong> as <strong>&lsquo;red&rsquo;</strong>, <strong><em>clouds</em></strong> as <strong>&lsquo;blue&rsquo;</strong> and <strong><em>unknowns</em></strong> as <strong>&lsquo;green&rsquo;</strong></td>\n    </tr>\n  </tbody>\n</table>\n</div>"}]},"apps":[],"jobName":"paragraph_1511371747469_1572618019","id":"20171020-070710_1843650237","dateCreated":"2017-11-22T17:29:07+0000","dateStarted":"2017-11-24T06:24:55+0000","dateFinished":"2017-11-24T06:24:55+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:6539"},{"text":"%md\n\n### Imports","user":"anonymous","dateUpdated":"2017-11-24T06:12:25+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Imports</h3>\n</div>"}]},"apps":[],"jobName":"paragraph_1511371747471_1061297067","id":"20171120-143714_472543342","dateCreated":"2017-11-22T17:29:07+0000","dateStarted":"2017-11-24T06:12:26+0000","dateFinished":"2017-11-24T06:12:26+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:6540"},{"text":"import java.io.File\r\nimport java.net.URL\r\n\r\nimport org.apache.commons.io.FileUtils\r\nimport org.datavec.api.records.reader.impl.csv.CSVRecordReader\r\nimport org.datavec.api.split.FileSplit\r\nimport org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator\r\nimport org.deeplearning4j.eval.{Evaluation, ROC}\r\nimport org.deeplearning4j.nn.api.{Model, OptimizationAlgorithm}\r\nimport org.deeplearning4j.nn.conf.{NeuralNetConfiguration, Updater}\r\nimport org.deeplearning4j.nn.conf.layers.{DenseLayer, OutputLayer}\r\nimport org.deeplearning4j.nn.multilayer.MultiLayerNetwork\r\nimport org.deeplearning4j.nn.weights.WeightInit\r\nimport org.deeplearning4j.optimize.api.IterationListener\r\nimport org.nd4j.linalg.activations.Activation\r\nimport org.nd4j.linalg.lossfunctions.LossFunctions.LossFunction","user":"anonymous","dateUpdated":"2017-11-24T06:12:26+0000","config":{"tableHide":true,"editorSetting":{"language":"scala","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/scala","fontSize":9,"editorHide":false,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"import java.io.File\nimport java.net.URL\nimport org.apache.commons.io.FileUtils\nimport org.datavec.api.records.reader.impl.csv.CSVRecordReader\nimport org.datavec.api.split.FileSplit\nimport org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator\nimport org.deeplearning4j.eval.{Evaluation, ROC}\nimport org.deeplearning4j.nn.api.{Model, OptimizationAlgorithm}\nimport org.deeplearning4j.nn.conf.{NeuralNetConfiguration, Updater}\nimport org.deeplearning4j.nn.conf.layers.{DenseLayer, OutputLayer}\nimport org.deeplearning4j.nn.multilayer.MultiLayerNetwork\nimport org.deeplearning4j.nn.weights.WeightInit\nimport org.deeplearning4j.optimize.api.IterationListener\nimport org.nd4j.linalg.activations.Activation\nimport org.nd4j.linalg.lossfunctions.LossFunctions.LossFunction\n"}]},"apps":[],"jobName":"paragraph_1511371747471_1254622651","id":"20171020-071303_1517144370","dateCreated":"2017-11-22T17:29:07+0000","dateStarted":"2017-11-24T06:12:26+0000","dateFinished":"2017-11-24T06:12:28+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:6541"},{"text":"%md\n\n### Creating the network","user":"anonymous","dateUpdated":"2017-11-24T06:12:26+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Creating the network</h3>\n</div>"}]},"apps":[],"jobName":"paragraph_1511371747473_-311153185","id":"20171020-072208_966782035","dateCreated":"2017-11-22T17:29:07+0000","dateStarted":"2017-11-24T06:12:28+0000","dateFinished":"2017-11-24T06:12:28+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:6542"},{"text":"val crossValSet = \"1\" // You can use this to specify (1-9) which of the train/test split you want to use.\r\n\r\nval seed = 123\r\nval learningRate = 0.008\r\nval batchSize = 32\r\nval nEpochs = 10\r\nval numInputs = 8\r\nval numOutputs = 2\r\nval numHiddenNodes = 50\r\nval baseUrl = \"https://raw.githubusercontent.com/bpark738/Cloud/master/src/main/resources\"\r\nval trainFileUrl = baseUrl + \"/train/\" + crossValSet + \".csv\"\r\nval testFileUrl = baseUrl + \"/test/\" + crossValSet + \".csv\"\r\n\r\nval trainFile: File = new File(\"train.csv\")\r\nval testFile: File = new File(\"test.csv\")\r\n\r\nFileUtils.copyURLToFile(new URL(trainFileUrl), trainFile)\r\nFileUtils.copyURLToFile(new URL(testFileUrl), testFile)\r\n\r\nval rrTrain = new CSVRecordReader(1)\r\nrrTrain.initialize(new FileSplit(trainFile))\r\nval trainIter = new RecordReaderDataSetIterator(rrTrain, batchSize, 0, 2)\r\nval rrTest = new CSVRecordReader(1)\r\nrrTest.initialize(new FileSplit(testFile))\r\nval testIter = new RecordReaderDataSetIterator(rrTest, batchSize, 0, 2)\r\nval conf = new NeuralNetConfiguration.Builder()\r\n    .seed(seed)\r\n    .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)\r\n    .learningRate(learningRate)\r\n    .updater(Updater.ADAM)\r\n    .list\r\n    .layer(0, new DenseLayer.Builder().nIn(numInputs).nOut(numHiddenNodes).weightInit(WeightInit.XAVIER).activation(Activation.RELU).build)\r\n    .layer(1, new OutputLayer.Builder(LossFunction.MCXENT).weightInit(WeightInit.XAVIER).activation(Activation.SOFTMAX).nIn(numHiddenNodes).nOut(numOutputs).build)\r\n    .pretrain(true).backprop(true)\r\n    .build\r\n    \r\nval model = new MultiLayerNetwork(conf)\r\nmodel.setListeners(new IterationListener {\r\n  override def invoke(): Unit = ???   \r\n  override def iterationDone(model: Model, iteration: Int): Unit = {\r\n    if(iteration % 2500 == 0) {\r\n      println(\"Score at iteration \" + iteration + \" is \" + model.score())\r\n    }\r\n  }   \r\n  override def invoked(): Nothing = ???\r\n})","user":"anonymous","dateUpdated":"2017-11-24T06:25:50+0000","config":{"tableHide":true,"editorSetting":{"language":"scala"},"colWidth":12,"editorMode":"ace/mode/scala","fontSize":9,"editorHide":false,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"seed: Int = 123\nlearningRate: Double = 0.008\nbatchSize: Int = 32\nnEpochs: Int = 10\nnumInputs: Int = 8\nnumOutputs: Int = 2\nnumHiddenNodes: Int = 50\ncrossValSet: String = 1\nbaseUrl: String = https://raw.githubusercontent.com/bpark738/Cloud/master/src/main/resources\ntrainFileUrl: String = https://raw.githubusercontent.com/bpark738/Cloud/master/src/main/resources/train/1.csv\ntestFileUrl: String = https://raw.githubusercontent.com/bpark738/Cloud/master/src/main/resources/test/1.csv\ntrainFile: java.io.File = train.csv\ntestFile: java.io.File = test.csv\nrrTrain: org.datavec.api.records.reader.impl.csv.CSVRecordReader = org.datavec.api.records.reader.impl.csv.CSVRecordReader@495da255\ntrainIter: org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator = org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator@754c0465\nrrTest: org.datavec.api.records.reader.impl.csv.CSVRecordReader = org.datavec.api.records.reader.impl.csv.CSVRecordReader@4e1014d6\ntestIter: org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator = org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator@5f62d6e9\nconf: org.deeplearning4j.nn.conf.MultiLayerConfiguration =\n{\n  \"backprop\" : true,\n  \"backpropType\" : \"Standard\",\n  \"cacheMode\" : \"NONE\",\n  \"confs\" : [ {\n    \"cacheMode\" : \"NONE\",\n    \"iterationCount\" : 0,\n    \"l1ByParam\" : { },\n    \"l2ByParam\" : { },\n    \"layer\" : {\n      \"dense\" : {\n        \"activationFn\" : {\n          \"ReLU\" : { }\n        },\n        \"adamMeanDecay\" : 0.9,\n        \"adamVarDecay\" : 0.999,\n        \"biasInit\" : 0.0,\n        \"biasLearningRate\" : 0.008,\n        \"dist\" : null,\n        \"dropOut\" : 0.0,\n        \"epsilon\" : 1.0E-8,\n        \"gradientNormalization\" : \"None\",\n        \"gradientNormalizationThreshold\" : 1.0,\n        \"iupdater\" : {\n          \"@class\" : \"org.nd4j.linalg.learning.config.Adam\",\n          \"beta1\" : 0.9,\n          \"beta2\" : 0.999,\n          \"epsilon\" : 1....model: org.deeplearning4j.nn.multilayer.MultiLayerNetwork = org.deeplearning4j.nn.multilayer.MultiLayerNetwork@6c33bfb0\n"}]},"apps":[],"jobName":"paragraph_1511371747473_-1559909459","id":"20171020-071349_473511535","dateCreated":"2017-11-22T17:29:07+0000","dateStarted":"2017-11-24T06:12:28+0000","dateFinished":"2017-11-24T06:13:18+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:6543"},{"text":"%md\n\n### Training and Evaluation","user":"anonymous","dateUpdated":"2017-11-24T06:22:33+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Training and Evaluation</h3>\n</div>"}]},"apps":[],"jobName":"paragraph_1511371747473_-1837881679","id":"20171121-190107_1061328323","dateCreated":"2017-11-22T17:29:07+0000","dateStarted":"2017-11-24T06:22:33+0000","dateFinished":"2017-11-24T06:22:33+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:6544"},{"text":"(1 to nEpochs).foreach((epoch) => {\r\n  println(\"Epoch number: \" + epoch)\r\n  model.fit(trainIter)\r\n})\r\n\r\nprintln(\"\\nEvaluate model....\")\r\nprintln(\"Model evaluation stats:\" + model.evaluate(testIter).stats(true))\r\ntestIter.reset()","user":"anonymous","dateUpdated":"2017-11-24T06:25:57+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":true},"editorMode":"ace/mode/scala","editorHide":false,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Epoch number: 1\nScore at iteration 0 is 11.265636444091797\nScore at iteration 2500 is 0.37792307138442993\nScore at iteration 5000 is 0.4590008556842804\nEpoch number: 2\nScore at iteration 7500 is 0.38077807426452637\nScore at iteration 10000 is 0.24501022696495056\nEpoch number: 3\nScore at iteration 12500 is 0.29872578382492065\nScore at iteration 15000 is 0.14576759934425354\nEpoch number: 4\nScore at iteration 17500 is 0.17461033165454865\nScore at iteration 20000 is 0.2820994257926941\nEpoch number: 5\nScore at iteration 22500 is 0.26163771748542786\nScore at iteration 25000 is 0.1494707316160202\nEpoch number: 6\nScore at iteration 27500 is 0.5326113700866699\nScore at iteration 30000 is 0.2801351845264435\nEpoch number: 7\nScore at iteration 32500 is 0.3669814467430115\nScore at iteration 35000 is 0.27670201659202576\nEpoch number: 8\nScore at iteration 37500 is 0.10141701996326447\nScore at iteration 40000 is 0.13498355448246002\nEpoch number: 9\nScore at iteration 42500 is 0.23960667848587036\nScore at iteration 45000 is 0.33983534574508667\nEpoch number: 10\nScore at iteration 47500 is 0.1606469452381134\nScore at iteration 50000 is 0.28919029235839844\n\nEvaluate model....\nModel evaluation stats:\nExamples labeled as 0 classified by model as 0: 2032 times\nExamples labeled as 0 classified by model as 1: 5021 times\nExamples labeled as 1 classified by model as 0: 439 times\nExamples labeled as 1 classified by model as 1: 40853 times\n\n\n==========================Scores========================================\n # of classes:    2\n Accuracy:        0.8871\n Precision:       0.8564\n Recall:          0.6387\n F1 Score:        0.9374\n========================================================================\n"}]},"apps":[],"jobName":"paragraph_1511441688574_-2000196616","id":"20171123-125448_2054357135","dateCreated":"2017-11-23T12:54:48+0000","dateStarted":"2017-11-24T06:13:18+0000","dateFinished":"2017-11-24T06:14:41+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:6545"},{"text":"%md\n\n## Visualizing the results","user":"anonymous","dateUpdated":"2017-11-24T06:12:26+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1511442250102_1640472443","id":"20171123-130410_171409456","dateCreated":"2017-11-23T13:04:10+0000","dateStarted":"2017-11-24T06:14:41+0000","dateFinished":"2017-11-24T06:14:41+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:6546","errorMessage":""},{"text":"val roc = new ROC(100)\r\nwhile (testIter.hasNext) {\r\n  val batch = testIter.next\r\n  val output = model.output(batch.getFeatures)\r\n  roc.eval(batch.getLabels, output)\r\n}\r\ntestIter.reset()","user":"anonymous","dateUpdated":"2017-11-24T06:26:51+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":true},"editorMode":"ace/mode/scala","editorHide":false,"tableHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"roc: org.deeplearning4j.eval.ROC = ROC(thresholdSteps=100, countActualPositive=0, countActualNegative=0, counts={0.0=ROC.CountsForThreshold(threshold=0.0, countTruePositive=0, countFalsePositive=0), 0.01=ROC.CountsForThreshold(threshold=0.01, countTruePositive=0, countFalsePositive=0), 0.02=ROC.CountsForThreshold(threshold=0.02, countTruePositive=0, countFalsePositive=0), 0.03=ROC.CountsForThreshold(threshold=0.03, countTruePositive=0, countFalsePositive=0), 0.04=ROC.CountsForThreshold(threshold=0.04, countTruePositive=0, countFalsePositive=0), 0.05=ROC.CountsForThreshold(threshold=0.05, countTruePositive=0, countFalsePositive=0), 0.06=ROC.CountsForThreshold(threshold=0.06, countTruePositive=0, countFalsePositive=0), 0.07=ROC.CountsForThreshold(threshold=0.07, countTruePositive=0, count..."}]},"apps":[],"jobName":"paragraph_1511443242174_479353050","id":"20171123-132042_689616490","dateCreated":"2017-11-23T13:20:42+0000","dateStarted":"2017-11-24T06:26:32+0000","dateFinished":"2017-11-24T06:26:45+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:6547"},{"text":"%md\n\n#### FINAL TEST Area Under the Curve (AUC)","user":"anonymous","dateUpdated":"2017-11-24T06:12:53+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h4>FINAL TEST Area Under the Curve (AUC)</h4>\n</div>"}]},"apps":[],"jobName":"paragraph_1511443424841_-1815971639","id":"20171123-132344_1485892742","dateCreated":"2017-11-23T13:23:44+0000","dateStarted":"2017-11-24T06:14:43+0000","dateFinished":"2017-11-24T06:14:43+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:6548"},{"text":"println(\"\\nFINAL TEST AUC: \" + roc.calculateAUC)","user":"anonymous","dateUpdated":"2017-11-24T06:26:55+0000","config":{"editorSetting":{"language":"scala","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/scala","fontSize":9,"results":{},"enabled":true,"editorHide":false,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"\nFINAL TEST AUC: 0.8523157647432149\n"}]},"apps":[],"jobName":"paragraph_1511371747474_-403657551","id":"20171121-190120_608905762","dateCreated":"2017-11-22T17:29:07+0000","dateStarted":"2017-11-24T06:26:53+0000","dateFinished":"2017-11-24T06:26:53+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:6549"},{"text":"%md\n\n## ROC plot","user":"anonymous","dateUpdated":"2017-11-24T06:12:26+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h2>ROC plot</h2>\n</div>"}]},"apps":[],"jobName":"paragraph_1511444443513_-1591780149","id":"20171123-134043_1668579252","dateCreated":"2017-11-23T13:40:43+0000","dateStarted":"2017-11-24T06:14:44+0000","dateFinished":"2017-11-24T06:14:44+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:6550"},{"text":"val df = sc.parallelize(roc.getRocCurve.getFpr zip roc.getRocCurve.getTpr).toDF(\"FPR\",\"TPR\")\ndf.registerTempTable(\"roc\")","user":"anonymous","dateUpdated":"2017-11-24T06:27:08+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":false},"editorMode":"ace/mode/scala","tableHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"df: org.apache.spark.sql.DataFrame = [FPR: double, TPR: double]\nwarning: there was one deprecation warning; re-run with -deprecation for details\n"}]},"apps":[],"jobName":"paragraph_1511444461422_-1200331485","id":"20171123-134101_1283686241","dateCreated":"2017-11-23T13:41:01+0000","dateStarted":"2017-11-24T06:27:04+0000","dateFinished":"2017-11-24T06:27:04+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:6551"},{"user":"anonymous","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{"0":{"graph":{"mode":"stackedAreaChart","height":300,"optionOpen":false,"setting":{"table":{"tableGridState":{},"tableColumnTypeState":{"names":{"FPR":"string","TPR":"string"},"updated":false},"tableOptionSpecHash":"[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer for displaying aggregated values\"}]","tableOptionValue":{"useFilter":false,"showPagination":false,"showAggregationFooter":false},"updated":false,"initialized":false},"stackedAreaChart":{"rotate":{"degree":"-45"},"xLabelStatus":"default","style":"stack"},"lineChart":{"rotate":{"degree":"-45"},"xLabelStatus":"default"}},"commonSetting":{},"keys":[{"name":"FPR","index":0,"aggr":"sum"}],"groups":[],"values":[{"name":"TPR","index":1,"aggr":"sum"}]},"helium":{}}},"editorSetting":{"language":"sql"},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1511450366704_1172498966","id":"20171123-151926_1944526401","dateCreated":"2017-11-23T15:19:26+0000","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:8287","text":"%sql\nselect FPR, TPR from roc order by FPR","dateUpdated":"2017-11-24T06:27:10+0000","dateFinished":"2017-11-24T06:27:09+0000","dateStarted":"2017-11-24T06:27:09+0000","results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"FPR\tTPR\n0.0\t0.0\n0.07160073727491847\t0.3035454809648358\n0.098539628526868\t0.4048726145500339\n0.11229264142917907\t0.4820304175142885\n0.12165036154827732\t0.5429380993897123\n0.1305827307528711\t0.5813716942749201\n0.13937331631929675\t0.6078174949142691\n0.14759676733304977\t0.6308728082921631\n0.15553665107046646\t0.6521117892085634\n0.16390188572238765\t0.674924924924925\n0.17240890401247697\t0.6987309890535697\n0.17935630228271657\t0.724232296812942\n0.1878633205728059\t0.7490070715877167\n0.19792995888274492\t0.7712874164487068\n0.20672054444917057\t0.7941489876973747\n0.21863037005529562\t0.8167683812845103\n0.2272791719835531\t0.8351012302625206\n0.23791294484616476\t0.8512544802867383\n0.2501063377286261\t0.8667538506248184\n0.261874379696583\t0.8789838225322096\n0.27264993619736283\t0.8901724304950112\n0.2851268963561605\t0.8999806257870774\n0.3017155820218347\t0.9079240530853434\n0.3151850276478094\t0.9153346895282379\n0.32936339146462495\t0.9212922600019374\n0.34283283709059975\t0.9272740482417902\n0.35616049907840636\t0.9320933837062869\n0.371331348362399\t0.9364041460815654\n0.38706933219906425\t0.940521166327618\n0.40379980150290656\t0.9445655332752106\n0.42038848716858074\t0.9490216022474087\n0.43442506734722813\t0.9527027027027027\n0.4495959166312208\t0.9557299234718589\n0.46604281865872677\t0.9592899350963867\n0.4802211824755423\t0.9623898091640027\n0.4956755990358713\t0.9653201588685459\n0.5118389337870409\t0.9679598953792502\n0.5272933503473699\t0.9701152765668895\n0.5421806323550262\t0.972488617649908\n0.5577768325535233\t0.9742565145790952\n0.5737983836665249\t0.9760970648067422\n0.5915213384375443\t0.9775985663082437\n0.6061250531688643\t0.9788578901482128\n0.6200198497093435\t0.9801898672866415\n0.6317878916773004\t0.9819335464496755\n0.64412306819793\t0.9836045723142497\n0.657876081100241\t0.9851060738157512\n0.670636608535375\t0.9864138331880268\n0.683397135970509\t0.9874309793664633\n0.6985679852545016\t0.988278601181827\n0.7118956472423082\t0.9893684006587232\n0.7236636892102651\t0.9902402402402403\n0.7347228129873813\t0.9911605153540637\n0.7474833404225153\t0.991862830572508\n0.7612363533248263\t0.9926135813232587\n0.7720119098256061\t0.9932916787755497\n0.7844888699844038\t0.9940182117601473\n0.7917198355309797\t0.9945267848493655\n0.802920742946264\t0.9949384868739708\n0.8121366794271941\t0.9955197132616488\n0.8217779668226287\t0.9958103264554877\n0.8307103360272224\t0.9960767218831735\n0.8382248688501347\t0.996415770609319\n0.8450304834822061\t0.996706383803158\n0.8518360981142776\t0.9971180858277633\n0.8590670636608535\t0.9973602634892957\n0.8664398128455976\t0.9975297878523685\n0.8735289947540054\t0.997771965513901\n0.8818942294059265\t0.9979414898769737\n0.8901176804196795\t0.998159449772353\n0.8970650786899191\t0.9983047563692725\n0.9035871260456543\t0.9983289741354258\n0.9095420388487169\t0.9984984984984985\n0.9166312207571247\t0.9985711517969582\n0.9225861335601871\t0.9987164583938778\n0.9274067772579044\t0.9988859827569505\n0.9344959591663122\t0.9990070715877167\n0.9403090883312066\t0.999128160418483\n0.9438536792854104\t0.9992008137169428\n0.947114702963278\t0.9992492492492493\n0.9529278321281724\t0.9993945558461688\n0.9571813412732171\t0.9994914269107817\n0.9608677158655891\t0.9995640802092415\n0.9637033886289522\t0.9996367335077012\n0.9688075996030058\t0.9996367335077012\n0.9727775414717141\t0.9996851690400077\n0.9756132142350773\t0.9996851690400077\n0.9784488869984403\t0.999709386806161\n0.9807174252091309\t0.9997578223384675\n0.9832695306961576\t0.9997820401046208\n0.9852545016305119\t0.999806257870774\n0.9870976889266979\t0.999806257870774\n0.9883737416702113\t0.999806257870774\n0.989933361690061\t0.9998546934030805\n0.9920601162625833\t0.9998789111692338\n0.9943286544732738\t0.9998789111692338\n0.9961718417694598\t0.999903128935387\n0.9977314617893095\t0.9999273467015403\n0.9995746490854955\t0.9999273467015403\n1.0\t1.0\n1.0\t0.9999757822338468\n"}]},"runtimeInfos":{"jobUrl":{"propertyName":"jobUrl","label":"SPARK JOB","tooltip":"View in Spark web UI","group":"spark","values":["http://172.17.0.2:4040/jobs/job?id=2"],"interpreterSettingId":"2CZQ7ZY6S"}}},{"text":"%md\n\n### Summary\n\nIn this tutorial, we learned about what cloud cover is and we trained a network in DL4J to determine cloud cover from a bunch of specified features. At the end, we did some visualizations from the results we obtained from the network.","user":"anonymous","dateUpdated":"2017-11-24T06:27:46+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Summary</h3>\n<p>In this tutorial, we learned about what cloud cover is and we trained a network in DL4J to determine cloud cover from a bunch of specified features. At the end, we did some visualizations from the results we obtained from the network.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1511371747474_-206179228","id":"20171117-073125_214586916","dateCreated":"2017-11-22T17:29:07+0000","dateStarted":"2017-11-24T06:27:46+0000","dateFinished":"2017-11-24T06:27:46+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:6553"},{"text":"%md\n\n### What's next?\n\n- Check out all of our tutorials available [on Github](https://github.com/deeplearning4j/deeplearning4j/tree/master/dl4j-examples/tutorials). Notebooks are numbered for easy following.","user":"anonymous","dateUpdated":"2017-11-24T06:12:27+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>What&rsquo;s next?</h3>\n<ul>\n  <li>Check out all of our tutorials available <a href=\"https://github.com/deeplearning4j/deeplearning4j/tree/master/dl4j-examples/tutorials\">on Github</a>. Notebooks are numbered for easy following.</li>\n</ul>\n</div>"}]},"apps":[],"jobName":"paragraph_1511371747475_-1176044095","id":"20171020-072151_195526063","dateCreated":"2017-11-22T17:29:07+0000","dateStarted":"2017-11-24T06:14:45+0000","dateFinished":"2017-11-24T06:14:45+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:6554"},{"text":"%md\n","user":"anonymous","dateUpdated":"2017-11-23T12:47:44+0000","config":{"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1511371747479_-1737549046","id":"20171020-072158_2072802023","dateCreated":"2017-11-22T17:29:07+0000","status":"FINISHED","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:6555"}],"name":"Determining cloud cover","id":"2CXYDWPFE","angularObjects":{"2CZQ7ZY6S:shared_process":[],"2D1CXP1PP:shared_process":[],"2CZ2UUD41:shared_process":[],"2D1AKPRZA:shared_process":[],"2CX1MZBK8:shared_process":[],"2CXVEN2YD:shared_process":[],"2CX4DY6NZ:shared_process":[],"2CXVDPAZN:shared_process":[],"2CXK9GBTB:shared_process":[],"2CXAT5DWH:shared_process":[],"2CXRBFGBY:shared_process":[],"2CZDYJPG9:shared_process":[],"2CY9RSS9K:shared_process":[],"2CZSZQC4M:shared_process":[],"2CYBYW659:shared_process":[],"2CZCD55JD:shared_process":[],"2CYAJA9NF:shared_process":[],"2D1FESVJD:shared_process":[],"2CYGUF5SA:shared_process":[],"2CZ4X76GG:shared_process":[]},"config":{"looknfeel":"default","personalizedMode":"false"},"info":{}}