{"paragraphs":[{"text":"%md\n### Note\n\nView the README.md [here](https://github.com/deeplearning4j/dl4j-examples/blob/master/tutorials/README.md) to learn about installing, setting up dependencies and importing notebooks in Zeppelin","user":"anonymous","dateUpdated":"2017-10-20T06:33:04+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true,"fontSize":9},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Note</h3>\n<p>View the README.md <a href=\"https://github.com/deeplearning4j/dl4j-examples/blob/master/tutorials/README.md\">here</a> to learn about installing, setting up dependencies and importing notebooks in Zeppelin</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1508479228592_1840193084","id":"20171016-071912_685216800","dateCreated":"2017-10-20T06:00:28+0000","dateStarted":"2017-10-20T06:33:04+0000","dateFinished":"2017-10-20T06:33:04+0000","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:25893"},{"text":"%md\n\n### Background\n---\n\nDeep learning is all about designing your network according to the data and its features that you already have - the dataset. So that the network can later train on it to get you the desired outcomes. There are different techniques to model your network and all of them build a relational hierarchy of some form between the inputs and outputs.\n\nIn this tutorial we are going to configure the simplest neural network and that is logistic regression model network. \n\nRegression is simply finding out the relations between the independant variables (inputs) and the dependant variables (outputs). A logistic regression is one in which the dependant variable is categorical - meaning that it can take only a limited number of values. For example, it can either be a cat or a dog, or it can be digits from 0-9.\n\n**A simple logisitic regression calculates 'x*w + b = y'. Where 'x' is our input, 'w' is the weight, 'b' is the bias and y is the output. Ignore the biological terms here (cell body, axon etc). Just focus on how the input flows towards the output**\n\n|---|---|---|\n|**Logistic Regression** | ![How a logistic regression is calculcated](https://i.pinimg.com/736x/61/fe/81/61fe81589ab491d1d3ba612b3bdf5b51--convolutional-neural-network-neuron-model.jpg) | [Source](https://i.pinimg.com/736x/61/fe/81/61fe81589ab491d1d3ba612b3bdf5b51--convolutional-neural-network-neuron-model.jpg) |","user":"anonymous","dateUpdated":"2017-10-20T06:52:44+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true,"fontSize":9},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Background</h3>\n<hr/>\n<p>Deep learning is all about designing your network according to the data and its features that you already have - the dataset. So that the network can later train on it to get you the desired outcomes. There are different techniques to model your network and all of them build a relational hierarchy of some form between the inputs and outputs.</p>\n<p>In this tutorial we are going to configure the simplest neural network and that is logistic regression model network. </p>\n<p>Regression is simply finding out the relations between the independant variables (inputs) and the dependant variables (outputs). A logistic regression is one in which the dependant variable is categorical - meaning that it can take only a limited number of values. For example, it can either be a cat or a dog, or it can be digits from 0-9.</p>\n<p><strong>A simple logisitic regression calculates &lsquo;x*w + b = y&rsquo;. Where &lsquo;x&rsquo; is our input, &lsquo;w&rsquo; is the weight, &lsquo;b&rsquo; is the bias and y is the output. Ignore the biological terms here (cell body, axon etc). Just focus on how the input flows towards the output</strong></p>\n<table>\n  <tbody>\n    <tr>\n      <td><strong>Logistic Regression</strong> </td>\n      <td><img src=\"https://i.pinimg.com/736x/61/fe/81/61fe81589ab491d1d3ba612b3bdf5b51--convolutional-neural-network-neuron-model.jpg\" alt=\"How a logistic regression is calculcated\" /> </td>\n      <td><a href=\"https://i.pinimg.com/736x/61/fe/81/61fe81589ab491d1d3ba612b3bdf5b51--convolutional-neural-network-neuron-model.jpg\">Source</a> </td>\n    </tr>\n  </tbody>\n</table>\n</div>"}]},"apps":[],"jobName":"paragraph_1508479228609_963668545","id":"20171016-074313_1713953229","dateCreated":"2017-10-20T06:00:28+0000","dateStarted":"2017-10-20T06:52:44+0000","dateFinished":"2017-10-20T06:52:45+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:25894"},{"text":"%md\n\n### What are we going to learn in this tutorial?\nAs Explained above, we're going to configure the most simplest network with only one input layer and one output layer to demostrate the concept of logistic regression.","user":"anonymous","dateUpdated":"2017-10-20T06:33:29+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true,"fontSize":9},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>What are we going to learn in this tutorial?</h3>\n<p>As Explained above, we&rsquo;re going to configure the most simplest network with only one input layer and one output layer to demostrate the concept of logistic regression.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1508479228610_660259320","id":"20171016-074409_2022759097","dateCreated":"2017-10-20T06:00:28+0000","dateStarted":"2017-10-20T06:33:29+0000","dateFinished":"2017-10-20T06:33:29+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:25896"},{"text":"%md\n\n### Imports","user":"anonymous","dateUpdated":"2017-10-20T06:33:04+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true,"fontSize":9},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Imports</h3>\n</div>"}]},"apps":[],"jobName":"paragraph_1508479228610_142406103","id":"20171016-080237_395422951","dateCreated":"2017-10-20T06:00:28+0000","dateStarted":"2017-10-20T06:33:09+0000","dateFinished":"2017-10-20T06:33:09+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:25897"},{"text":"import org.deeplearning4j.nn.api.OptimizationAlgorithm\nimport org.deeplearning4j.nn.conf.graph.MergeVertex\nimport org.deeplearning4j.nn.conf.layers.{DenseLayer, GravesLSTM, OutputLayer, RnnOutputLayer}\nimport org.deeplearning4j.nn.conf.{ComputationGraphConfiguration, MultiLayerConfiguration, NeuralNetConfiguration}\nimport org.deeplearning4j.nn.graph.ComputationGraph\nimport org.deeplearning4j.nn.multilayer.MultiLayerNetwork\nimport org.deeplearning4j.nn.weights.WeightInit\nimport org.nd4j.linalg.activations.Activation\nimport org.nd4j.linalg.learning.config.Nesterovs\nimport org.nd4j.linalg.lossfunctions.LossFunctions","user":"anonymous","dateUpdated":"2017-10-20T06:33:05+0000","config":{"tableHide":true,"editorSetting":{"language":"scala","editOnDblClick":false},"colWidth":12,"editorMode":"ace/mode/scala","results":{},"enabled":true,"fontSize":9},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"import org.deeplearning4j.nn.api.OptimizationAlgorithm\nimport org.deeplearning4j.nn.conf.graph.MergeVertex\nimport org.deeplearning4j.nn.conf.layers.{DenseLayer, GravesLSTM, OutputLayer, RnnOutputLayer}\nimport org.deeplearning4j.nn.conf.{ComputationGraphConfiguration, MultiLayerConfiguration, NeuralNetConfiguration}\nimport org.deeplearning4j.nn.graph.ComputationGraph\nimport org.deeplearning4j.nn.multilayer.MultiLayerNetwork\nimport org.deeplearning4j.nn.weights.WeightInit\nimport org.nd4j.linalg.activations.Activation\nimport org.nd4j.linalg.learning.config.Nesterovs\nimport org.nd4j.linalg.lossfunctions.LossFunctions\n"}]},"apps":[],"jobName":"paragraph_1508479228610_1910058437","id":"20171016-080255_207308565","dateCreated":"2017-10-20T06:00:28+0000","dateStarted":"2017-10-20T06:33:09+0000","dateFinished":"2017-10-20T06:33:10+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:25898"},{"text":"%md\n\n### Configuring our logistic regression layers\nWe are going to first build our layers and then feed these layers into our network configuration.","user":"anonymous","dateUpdated":"2017-10-20T06:33:05+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true,"fontSize":9},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Configuring our logistic regression layers</h3>\n<p>We are going to first build our layers and then feed these layers into our network configuration.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1508479228611_-1906048339","id":"20171016-080332_621979098","dateCreated":"2017-10-20T06:00:28+0000","dateStarted":"2017-10-20T06:33:10+0000","dateFinished":"2017-10-20T06:33:10+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:25899"},{"text":"//Building the output layer\nval outputLayer : OutputLayer = new OutputLayer.Builder()\n    .nIn(784) //The number of inputs feed from the input layer\n    .nOut(10) //The number of output values the output layer is supposed to take\n    .weightInit(WeightInit.XAVIER) //The algorithm to use for weights initialization\n    .activation(Activation.SOFTMAX) //Softmax activate converts the output layer into a probability distribution\n    .build() //Building our output layer","user":"anonymous","dateUpdated":"2017-10-20T06:33:05+0000","config":{"tableHide":true,"editorSetting":{"language":"scala","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/scala","editorHide":false,"results":{},"enabled":true,"fontSize":9},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"outputLayer: org.deeplearning4j.nn.conf.layers.OutputLayer = OutputLayer(super=BaseOutputLayer(super=FeedForwardLayer(super=BaseLayer(activationFn=softmax, weightInit=XAVIER, biasInit=NaN, dist=null, learningRate=NaN, biasLearningRate=NaN, learningRateSchedule=null, momentum=NaN, momentumSchedule=null, l1=NaN, l2=NaN, l1Bias=NaN, l2Bias=NaN, updater=null, iUpdater=null, rho=NaN, epsilon=NaN, rmsDecay=NaN, adamMeanDecay=NaN, adamVarDecay=NaN, gradientNormalization=null, gradientNormalizationThreshold=NaN), nIn=784, nOut=10), lossFn=LossMCXENT()))\n"}]},"apps":[],"jobName":"paragraph_1508479228611_2062552918","id":"20171016-080655_2014399759","dateCreated":"2017-10-20T06:00:28+0000","dateStarted":"2017-10-20T06:33:10+0000","dateFinished":"2017-10-20T06:33:11+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:25900"},{"text":"//Since this is a simple network with a stack of layers we're going to configure a MultiLayerNetwork\nval logisticRegressionConf : MultiLayerConfiguration = new NeuralNetConfiguration.Builder()\n    .seed(123).learningRate(0.1).iterations(1).optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT).updater(new Nesterovs(0.9)) //High Level Configuration\n    .list() //For configuring MultiLayerNetwork we call the list method\n    .layer(0, outputLayer) //    <----- output layer fed here\n    .pretrain(false).backprop(true) //Pretraining and Backprop Configuration\n    .build() //Building Configuration","user":"anonymous","dateUpdated":"2017-10-20T06:33:05+0000","config":{"tableHide":true,"editorSetting":{"language":"scala","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/scala","editorHide":false,"results":{},"enabled":true,"fontSize":9},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"logisticRegressionConf: org.deeplearning4j.nn.conf.MultiLayerConfiguration =\n{\n  \"backprop\" : true,\n  \"backpropType\" : \"Standard\",\n  \"cacheMode\" : \"NONE\",\n  \"confs\" : [ {\n    \"cacheMode\" : \"NONE\",\n    \"iterationCount\" : 0,\n    \"l1ByParam\" : { },\n    \"l2ByParam\" : { },\n    \"layer\" : {\n      \"output\" : {\n        \"activationFn\" : {\n          \"Softmax\" : { }\n        },\n        \"adamMeanDecay\" : \"NaN\",\n        \"adamVarDecay\" : \"NaN\",\n        \"biasInit\" : 0.0,\n        \"biasLearningRate\" : 0.1,\n        \"dist\" : null,\n        \"dropOut\" : 0.0,\n        \"epsilon\" : \"NaN\",\n        \"gradientNormalization\" : \"None\",\n        \"gradientNormalizationThreshold\" : 1.0,\n        \"iupdater\" : {\n          \"@class\" : \"org.nd4j.linalg.learning.config.Nesterovs\",\n          \"learningRate\" : 0.1,\n          \"momentu..."}]},"apps":[],"jobName":"paragraph_1508479228611_1317000066","id":"20171016-080400_1846176330","dateCreated":"2017-10-20T06:00:28+0000","dateStarted":"2017-10-20T06:33:11+0000","dateFinished":"2017-10-20T06:33:11+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:25901"},{"text":"%md\n\n**This is how our configuration here looks like:**\n\n|---|---|---|\n|**Logistic Regression** | ![How a logistic regression visually looks like in neural networks](https://isaacchanghau.github.io/images/deeplearning/activationfunction/softmax.png) | [Source](https://isaacchanghau.github.io/images/deeplearning/activationfunction/softmax.png) |\n---\nThe layer with x1, x2, x3, ..., xn is out input layer. While the one with z1, z2, z3, ..., zk is our output layer. See how the weights and biases are connected and how softmax is applied to give the probability distribution.\n","user":"anonymous","dateUpdated":"2017-10-20T06:53:16+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true,"fontSize":9},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<p><strong>This is how our configuration here looks like:</strong></p>\n<table>\n  <tbody>\n    <tr>\n      <td><strong>Logistic Regression</strong> </td>\n      <td><img src=\"https://isaacchanghau.github.io/images/deeplearning/activationfunction/softmax.png\" alt=\"How a logistic regression visually looks like in neural networks\" /> </td>\n      <td><a href=\"https://isaacchanghau.github.io/images/deeplearning/activationfunction/softmax.png\">Source</a> </td>\n    </tr>\n  </tbody>\n</table>\n<hr/>\n<p>The layer with x1, x2, x3, &hellip;, xn is out input layer. While the one with z1, z2, z3, &hellip;, zk is our output layer. See how the weights and biases are connected and how softmax is applied to give the probability distribution.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1508479228611_-129871218","id":"20171016-083206_982462751","dateCreated":"2017-10-20T06:00:28+0000","dateStarted":"2017-10-20T06:53:16+0000","dateFinished":"2017-10-20T06:53:16+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:25902"},{"text":"%md\n\n### Why didn't we build our input layer\n---\nYou must be wondering why didn't we write any kind of code for building our input layer.That's because the input layer is only a set of inputs values fed into the network. It doesn't perform any kind of calculation. It's just an input sequence (raw or pre-processed) coming into the network to be trained on or to be evaluated upon. Later on, we are going to work with data iterators, which are supposed to feed our input to our network in a specific pattern, which can be thought of as an input layer of the network.","user":"anonymous","dateUpdated":"2017-10-20T06:33:05+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true,"fontSize":9},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Why didn&rsquo;t we build our input layer</h3>\n<hr/>\n<p>You must be wondering why didn&rsquo;t we write any kind of code for building our input layer.That&rsquo;s because the input layer is only a set of inputs values fed into the network. It doesn&rsquo;t perform any kind of calculation. It&rsquo;s just an input sequence (raw or pre-processed) coming into the network to be trained on or to be evaluated upon. Later on, we are going to work with data iterators, which are supposed to feed our input to our network in a specific pattern, which can be thought of as an input layer of the network.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1508479228612_-1891886006","id":"20171016-081502_1957369762","dateCreated":"2017-10-20T06:00:28+0000","dateStarted":"2017-10-20T06:33:12+0000","dateFinished":"2017-10-20T06:33:12+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:25903"},{"text":"%md \n\n### What's next?\n- See tutorial [here](http://someurl) to learn about configuring a more complex network, namely, a 'feedforward neural network'. Here we also introduce the concept of hidden layers.","user":"anonymous","dateUpdated":"2017-10-20T06:33:05+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true,"fontSize":9},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>What&rsquo;s next?</h3>\n<ul>\n  <li>See tutorial <a href=\"http://someurl\">here</a> to learn about configuring a more complex network, namely, a &lsquo;feedforward neural network&rsquo;. Here we also introduce the concept of hidden layers.</li>\n</ul>\n</div>"}]},"apps":[],"jobName":"paragraph_1508479228612_-1766342584","id":"20171016-074036_117973210","dateCreated":"2017-10-20T06:00:28+0000","dateStarted":"2017-10-20T06:33:12+0000","dateFinished":"2017-10-20T06:33:12+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:25904"},{"text":"%md\n","user":"anonymous","dateUpdated":"2017-10-20T06:33:05+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown"},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1508479228615_727381989","id":"20171016-074241_2030295323","dateCreated":"2017-10-20T06:00:28+0000","status":"FINISHED","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:25905"}],"name":"Logistic Regression","id":"2CW71ZY36","angularObjects":{"2CYUXYA5D:shared_process":[],"2CWKFWRMT:shared_process":[],"2CXBS1NKG:shared_process":[],"2CWGSEBDH:shared_process":[],"2CY5MAMP7:shared_process":[],"2CY8FT68P:shared_process":[],"2CYT5E816:shared_process":[],"2CVKF762K:shared_process":[],"2CVFFNGQ1:shared_process":[],"2CVXT54KH:shared_process":[],"2CWNUX3BY:shared_process":[],"2CVKHQX96:shared_process":[],"2CWYXKD9P:shared_process":[],"2CVJR5WZH:shared_process":[],"2CVFW4QC1:shared_process":[],"2CWZFE3AE:shared_process":[],"2CYQ734ZP:shared_process":[],"2CVHTA1KV:shared_process":[],"2CXUYR18D:shared_process":[],"2CWVYEYGS:shared_process":[]},"config":{"looknfeel":"default","personalizedMode":"false"},"info":{}}