{"paragraphs":[{"text":"%md\n### Note\n\nPlease view the [README](https://github.com/deeplearning4j/dl4j-examples/blob/master/tutorials/README.md) to learn about installing, setting up dependencies, and importing notebooks in Zeppelin","user":"anonymous","dateUpdated":"2017-10-16T14:32:32-0700","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Note</h3>\n<p>Please view the <a href=\"https://github.com/deeplearning4j/dl4j-examples/blob/master/tutorials/README.md\">README</a> to learn about installing, setting up dependencies and importing notebooks in Zeppelin</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1508177767346_1529182728","id":"20171016-071912_685216800","dateCreated":"2017-10-16T11:16:07-0700","dateStarted":"2017-10-16T14:32:32-0700","dateFinished":"2017-10-16T14:32:32-0700","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:22025"},{"text":"%md\n\n### Background\n\nWhether you are just getting started with Deeplearning4j or have spent a lot of time researching deep learning, toy datasets are essential for testing hypotheses and getting started with any neural network training process. Deeplearning4j comes with built-in data iterators for common datasets, including but not limited to:\n\n- MNIST\n- Iris\n- TinyImageNet (subset of ImageNet)\n- CIFAR-10\n- Labelled Faces in the Wild\n- Curve Fragment Ground-Truth Dataset\n\nBefore deep learning caught on as the de facto standard for some machine learning problems, many of these datasets were also used as a baseline for testing other machine leanring algorithms. Note that before you decide to implement any of these datasets in your project, check that you are using its license correctly (for example, you must obtain special permission to use ImageNet in a commercial project).","user":"anonymous","dateUpdated":"2017-10-16T14:32:36-0700","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Background</h3>\n<p>Whether you are just getting started with Deeplearning4j or have spent a lot of time researching deep learning, toy datasets are essential for testing hypotheses and getting started with any neural network training process. Deeplearning4j comes with built-in data iterators for common datasets, including but not limited to:</p>\n<ul>\n  <li>MNIST</li>\n  <li>Iris</li>\n  <li>TinyImageNet (subset of ImageNet)</li>\n  <li>CIFAR-10</li>\n  <li>Labelled Faces in the Wild</li>\n  <li>Curve Fragment Ground-Truth Dataset</li>\n</ul>\n<p>Before deep learning caught on as the de facto standard for some machine learning problems, many of these datasets were also used as a baseline for testing other machine leanring algorithms. Note that before you decide to implement any of these datasets in your project, check that you are using its license correctly (for example, you must obtain special permission to use ImageNet in a commercial project).</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1508177767347_1528797979","id":"20171016-074313_1713953229","dateCreated":"2017-10-16T11:16:07-0700","dateStarted":"2017-10-16T14:32:36-0700","dateFinished":"2017-10-16T14:32:36-0700","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:22027"},{"text":"%md\n\n### What are we going to learn in this tutorial?\n\nBuilding on what has been learned previously with using MultiLayerNetwork and ComputationGraph, we will instantiate a couple data iterators to feed a toy dataset to a neural network and perform a basic training process. This tutorial is focused on classifier training (you can also train networks for regression or use them for unsupervised training via an autoencoder), and you will also learn how to interpret the output in the console.","user":"anonymous","dateUpdated":"2017-10-16T14:32:38-0700","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>What are we going to learn in this tutorial?</h3>\n<p>Building on what has been learned previously with using MultiLayerNetwork and ComputationGraph, we will instantiate a couple data iterators to feed a toy dataset to a neural network and perform a basic training process. This tutorial is focused on classifier training (you can also train networks for regression or use them for unsupervised training via an autoencoder), and you will also learn how to interpret the output in the console.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1508177767348_1526874235","id":"20171016-074409_2022759097","dateCreated":"2017-10-16T11:16:07-0700","dateStarted":"2017-10-16T14:32:38-0700","dateFinished":"2017-10-16T14:32:38-0700","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:22028"},{"text":"%md\n\n### Imports","user":"anonymous","dateUpdated":"2017-10-16T14:32:40-0700","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Imports</h3>\n</div>"}]},"apps":[],"jobName":"paragraph_1508177767348_1526874235","id":"20171016-080237_395422951","dateCreated":"2017-10-16T11:16:07-0700","dateStarted":"2017-10-16T14:32:40-0700","dateFinished":"2017-10-16T14:32:40-0700","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:22029"},{"text":"%spark\n\nimport org.nd4j.linalg.activations.Activation\nimport org.nd4j.linalg.dataset.api.iterator.DataSetIterator\nimport org.deeplearning4j.datasets.iterator.impl.MnistDataSetIterator\nimport org.deeplearning4j.eval.Evaluation\nimport org.deeplearning4j.nn.api.OptimizationAlgorithm\nimport org.deeplearning4j.nn.conf.MultiLayerConfiguration\nimport org.deeplearning4j.nn.conf.NeuralNetConfiguration\nimport org.deeplearning4j.nn.conf.Updater\nimport org.deeplearning4j.nn.conf.layers.DenseLayer\nimport org.deeplearning4j.nn.conf.layers.OutputLayer\nimport org.deeplearning4j.nn.multilayer.MultiLayerNetwork\nimport org.deeplearning4j.nn.weights.WeightInit\nimport org.deeplearning4j.optimize.listeners.ScoreIterationListener\nimport org.nd4j.linalg.api.ndarray.INDArray\nimport org.nd4j.linalg.dataset.DataSet\nimport org.nd4j.linalg.lossfunctions.LossFunctions.LossFunction\nimport org.slf4j.Logger\nimport org.slf4j.LoggerFactory","user":"anonymous","dateUpdated":"2017-10-16T13:19:21-0700","config":{"tableHide":true,"editorSetting":{"language":"scala","editOnDblClick":false},"colWidth":12,"editorMode":"ace/mode/scala","results":{},"enabled":true},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1508177767348_1526874235","id":"20171016-080255_207308565","dateCreated":"2017-10-16T11:16:07-0700","dateStarted":"2017-10-16T13:19:21-0700","dateFinished":"2017-10-16T13:19:28-0700","status":"FINISHED","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:22030"},{"text":"%md\n\n### The MNIST classifier network\n\nA MultiLayerNetwork is well-equipped to build a classifier for MNIST digits. If you are not familiar with MNIST, it is a dataset originally assembled for handwriting recognition. You can read more about MNIST [here](https://en.wikipedia.org/wiki/MNIST_database).\n\nOnce the appropriate imports are in scope, set up a basic MultiLayerNetwork like below.","user":"anonymous","dateUpdated":"2017-10-16T13:12:05-0700","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1508177767348_1526874235","id":"20171016-080332_621979098","dateCreated":"2017-10-16T11:16:07-0700","dateStarted":"2017-10-16T13:12:05-0700","dateFinished":"2017-10-16T13:12:05-0700","status":"FINISHED","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:22031"},{"text":"//number of rows and columns in the input pictures\nval numRows = 28\nval numColumns = 28\nval outputNum = 10 // number of output classes\nval batchSize = 128 // batch size for each epoch\nval rngSeed = 123 // random number seed for reproducibility\nval numEpochs = 15 // number of epochs to perform\n\nval conf: MultiLayerConfiguration = new NeuralNetConfiguration.Builder()\n    .seed(rngSeed) //include a random seed for reproducibility\n    .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT) // use stochastic gradient descent as an optimization algorithm\n    .iterations(1) // in nearly all cases should be 1\n    .learningRate(0.006) //specify the learning rate\n    .updater(Updater.NESTEROVS).momentum(0.9) //specify the rate of change of the learning rate.\n    .regularization(true).l2(1e-4)\n    .list()\n    .layer(0, new DenseLayer.Builder() //create the first, input layer with xavier initialization\n            .nIn(numRows * numColumns)\n            .nOut(1000)\n            .activation(Activation.RELU)\n            .weightInit(WeightInit.XAVIER)\n            .build())\n    .layer(1, new OutputLayer.Builder(LossFunction.NEGATIVELOGLIKELIHOOD) //create hidden layer\n            .nIn(1000)\n            .nOut(outputNum)\n            .activation(Activation.SOFTMAX)\n            .weightInit(WeightInit.XAVIER)\n            .build())\n    .build()\n\nval model = new MultiLayerNetwork(conf)\nmodel.init()\n//print the score with every 1 iteration\nmodel.setListeners(new ScoreIterationListener(1))","user":"anonymous","dateUpdated":"2017-10-16T14:21:16-0700","config":{"tableHide":true,"editorSetting":{"language":"scala","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/scala","editorHide":false,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1508177767349_1526489486","id":"20171016-080655_2014399759","dateCreated":"2017-10-16T11:16:07-0700","dateStarted":"2017-10-16T14:21:08-0700","dateFinished":"2017-10-16T14:21:08-0700","status":"FINISHED","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:22032"},{"text":"%md\n\n### Using the MNIST iterator\n\nThe MNIST iterator, like most of the Deeplearning4j built-in iterators, extends the `DataSetIterator` class. This API allows for simple instantiation of datasets and automatic downloading of data in the background. The MNIST data iterator API specifically allows you to specify whether you are using the training or testing dataset, so instantiate two different iterators so you can evaluate your network.","user":"anonymous","dateUpdated":"2017-10-16T14:32:44-0700","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Using the MNIST iterator</h3>\n<p>The MNIST iterator, like most of the Deeplearning4j built-in iterators, extends the <code>DataSetIterator</code> class. This API allows for simple instantiation of datasets and automatic downloading of data in the background. The MNIST data iterator API specifically allows you to specify whether you are using the training or testing dataset, so instantiate two different iterators so you can evaluate your network.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1508177767349_1526489486","id":"20171016-081502_1957369762","dateCreated":"2017-10-16T11:16:07-0700","dateStarted":"2017-10-16T14:32:44-0700","dateFinished":"2017-10-16T14:32:44-0700","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:22033"},{"text":"\n//Get the DataSetIterators:\nval mnistTrain = new MnistDataSetIterator(batchSize, true, rngSeed)\nval mnistTest = new MnistDataSetIterator(batchSize, false, rngSeed)","user":"anonymous","dateUpdated":"2017-10-16T14:21:28-0700","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":false},"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1508181116786_184461776","id":"20171016-121156_892212367","dateCreated":"2017-10-16T12:11:56-0700","dateStarted":"2017-10-16T14:21:28-0700","dateFinished":"2017-10-16T14:21:29-0700","status":"FINISHED","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:22034"},{"text":"%md \n\n### Performing basic training\n\nNow that the network configruation is set up and instantiated along with our MNIST test/train iterators, training takes just a few lines of code. The fun begins.\n\nEarlier we attached a `ScoreIterationListener` to the model by using the `setListeners()` method. Depending on the browser you are using to run this notebook, you can open the debugger/inspector to view listener output. This output is redirected to the console since the internals of Deeplearning4j use SL4J for logging, and the output is being redirected by Zeppelin. This is a good thing since it can reduce clutter in notebooks.\n\nAs the model continues to train, a well-tuned model's score will lower with each iteration. This will eventually converge to a value close to zero. Note that more complex networks and problems may never yield an optimal score, this is where you need to become the expert and continue to tune and change your model's configuration.","user":"anonymous","dateUpdated":"2017-10-16T14:32:45-0700","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Performing basic training</h3>\n<p>Now that the network configruation is set up and instantiated along with our MNIST test/train iterators, training takes just a few lines of code. The fun begins.</p>\n<p>Earlier we attached a <code>ScoreIterationListener</code> to the model by using the <code>setListeners()</code> method. Depending on the browser you are using to run this notebook, you can open the debugger/inspector to view listener output. This output is redirected to the console since the internals of Deeplearning4j use SL4J for logging, and the output is being redirected by Zeppelin. This is a good thing since it can reduce clutter in notebooks.</p>\n<p>As the model continues to train, a well-tuned model&rsquo;s score will lower with each iteration. This will eventually converge to a value close to zero. Note that more complex networks and problems may never yield an optimal score, this is where you need to become the expert and continue to tune and change your model&rsquo;s configuration.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1508177767350_1527643733","id":"20171016-074036_117973210","dateCreated":"2017-10-16T11:16:07-0700","dateStarted":"2017-10-16T14:32:45-0700","dateFinished":"2017-10-16T14:32:45-0700","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:22035"},{"text":"// the \"simple\" way to do multiple epochs is to wrap fit() in a loop\n(1 to numEpochs).foreach(_ => model.fit(mnistTrain) )","user":"anonymous","dateUpdated":"2017-10-16T14:32:48-0700","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":true},"editorMode":"ace/mode/scala","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"apps":[],"jobName":"paragraph_1508182903935_-2030610450","id":"20171016-124143_922252682","dateCreated":"2017-10-16T12:41:43-0700","dateStarted":"2017-10-16T14:32:48-0700","dateFinished":"2017-10-16T14:35:34-0700","status":"ABORT","progressUpdateIntervalMs":500,"$$hashKey":"object:22036"},{"text":"%md\n\n### Evaluating the model\n\n\"Overfitting\" is a common problem in deep learning where your model doesn't generalize well to the problem you are trying to solve. This can happen when you have run too many epochs on a training dataset, when you haven't used a technique like [Dropout](https://en.wikipedia.org/wiki/Dropout_%28neural_networks%29), or your training dataset isn't big enough and doesn't encapsulate all of the features that are descriptive of your classes.\n\nDeeplearning4j comes with a suite of built-in tools for evaluation. The simplest method is to pass a testing iterator to `eval()` and retrieve an `Evaluation` object. Many more tools are also available, including ROC plotting and regression evaluation, which are available in the [org.deeplearning4j.eval](https://github.com/deeplearning4j/deeplearning4j/tree/master/deeplearning4j-core/src/test/java/org/deeplearning4j/eval) package.","user":"anonymous","dateUpdated":"2017-10-16T14:32:56-0700","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Evaluating the model</h3>\n<p>&ldquo;Overfitting&rdquo; is a common problem in deep learning where your model doesn&rsquo;t generalize well to the problem you are trying to solve. This can happen when you have run too many epochs on a training dataset, when you haven&rsquo;t used a technique like <a href=\"https://en.wikipedia.org/wiki/Dropout_%28neural_networks%29\">Dropout</a>, or your training dataset isn&rsquo;t big enough and doesn&rsquo;t encapsulate all of the features that are descriptive of your classes.</p>\n<p>Deeplearning4j comes with a suite of built-in tools for evaluation. The simplest method is to pass a testing iterator to <code>eval()</code> and retrieve an <code>Evaluation</code> object. Many more tools are also available, including ROC plotting and regression evaluation, which are available in the <a href=\"https://github.com/deeplearning4j/deeplearning4j/tree/master/deeplearning4j-core/src/test/java/org/deeplearning4j/eval\">org.deeplearning4j.eval</a> package.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1508183410347_-793322573","id":"20171016-125010_1283635646","dateCreated":"2017-10-16T12:50:10-0700","dateStarted":"2017-10-16T14:32:56-0700","dateFinished":"2017-10-16T14:32:56-0700","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:22037"},{"text":"val evaluation = model.evaluate(mnistTest)\n\n// print the basic statistics about the trained classifier\nprintln(\"Accuracy: \"+evaluation.accuracy())\nprintln(\"Precision: \"+evaluation.precision())\nprintln(\"Recall: \"+evaluation.recall())\n\n// in more complex scenarios, a confusion matrix is quite helpful\nprintln(evaluation.confusionToString())","user":"anonymous","dateUpdated":"2017-10-16T14:32:58-0700","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":true},"editorMode":"ace/mode/scala","editorHide":false,"tableHide":true},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1508183998594_-20315537","id":"20171016-125958_2130833332","dateCreated":"2017-10-16T12:59:58-0700","dateStarted":"2017-10-16T14:25:01-0700","dateFinished":"2017-10-16T14:25:02-0700","status":"FINISHED","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:22038"},{"text":"%md\n\n### What's next?\n\n- Check out all of our tutorials available [on Github](https://github.com/deeplearning4j/dl4j-examples/tree/master/tutorials). Notebooks are numbered for easy following.","user":"anonymous","dateUpdated":"2017-10-16T17:34:29-0700","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>What&rsquo;s next?</h3>\n<ul>\n  <li>Check out all of our tutorials available <a href=\"https://github.com/deeplearning4j/dl4j-examples/tree/master/tutorials\">on Github</a>. Notebooks are numbered for easy following.</li>\n</ul>\n</div>"}]},"apps":[],"jobName":"paragraph_1508177767350_1527643733","id":"20171016-074109_800628849","dateCreated":"2017-10-16T11:16:07-0700","dateStarted":"2017-10-16T17:34:29-0700","dateFinished":"2017-10-16T17:34:29-0700","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:22039"},{"text":"%md","user":"anonymous","dateUpdated":"2017-10-16T13:12:06-0700","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"scala","editOnDblClick":true},"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1508177767350_1527643733","id":"20171016-074241_2030295323","dateCreated":"2017-10-16T11:16:07-0700","status":"FINISHED","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:22040"}],"name":"Built-in Data Iterators","id":"2CY22DZ1M","angularObjects":{"2CUFAXJH1:shared_process":[],"2CTTRCUW9:shared_process":[],"2CXFB1JD9:shared_process":[],"2CV9EBN8T:shared_process":[],"2CVMYA8NZ:shared_process":[],"2CU4G5D4H:shared_process":[],"2CV9AER6T:shared_process":[],"2CUVA2HFP:shared_process":[],"2CVUSRMB1:shared_process":[],"2CWVF7734:shared_process":[],"2CXF26E3B:shared_process":[],"2CVQPUV2Y:shared_process":[],"2CW5PSGMM:shared_process":[],"2CUC2U3B1:shared_process":[],"2CUA1YWFX:shared_process":[],"2CUAZG96F:shared_process":[],"2CUGDBR4Y:shared_process":[],"2CX69KKCH:shared_process":[],"2CVNXS7G2:shared_process":[]},"config":{"looknfeel":"default","personalizedMode":"false"},"info":{}}