{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# How to use learners\n", "\n", "In CNTK learners are implementations of gradient-based optimization algorithms. CNTK automatically computes the gradient of your criterion/loss with respect to each learnable parameter but how this gradient is combined with the current parameter value to provide a new parameter value is left to the learner. \n", "\n", "CNTK provides three ways to define your learner, which we describe in detail in this notebook. You can\n", "- use a builtin learner. Builtin learners are very fast.\n", "- define your learner as a CNTK expression. This is not as fast as the builtin learners but more flexible. \n", "- define your learner as a Python function. This is even more flexible but even less fast. \n", "\n", "Here's a \"hello world\" example for learners." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import cntk as C\n", "import numpy as np\n", "import math\n", "np.set_printoptions(precision=4)\n", "\n", "features = C.input_variable(3)\n", "label = C.input_variable(2)\n", "z = C.layers.Sequential([C.layers.Dense(4, activation=C.relu), C.layers.Dense(2)])(features)\n", "\n", "lr_schedule_m = C.learning_rate_schedule(0.5, C.UnitType.minibatch)\n", "lr_schedule_s = C.learning_rate_schedule(0.5, C.UnitType.sample)\n", "\n", "sgd_learner_m = C.sgd(z.parameters, lr_schedule_m)\n", "sgd_learner_s = C.sgd(z.parameters, lr_schedule_s)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We have created two learners here. When creating a learner we have to specify a learning rate schedule, which can be as simple as specifying a single number (0.5 in this example) or it can be a list of learning rates that specify what the learning rate should be at different points in time. \n", "\n", "Currently, the best results with deep learning are obtained by having a small number of *phases* where inside each phase the learning rate is fixed and the learning rate decays by a constant factor when moving from phase `i` to phase `i+1`. We will come back to this point later.\n", "\n", "The second parameter in the learning rate schedule can be one of two different value:\n", "- Per minibatch\n", "- Per sample\n", "\n", "To understand the difference and get familiar with the learner properties and methods, let's write a small function that inspects the effect of a learner on the parameters assuming the parameters are all 0 and the gradients are all 1." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "unit = sample\n", " [array([[-0.5, -0.5],\n", " [-0.5, -0.5],\n", " [-0.5, -0.5],\n", " [-0.5, -0.5]], dtype=float32), array([-0.5, -0.5], dtype=float32), array([[-0.5, -0.5, -0.5, -0.5],\n", " [-0.5, -0.5, -0.5, -0.5],\n", " [-0.5, -0.5, -0.5, -0.5]], dtype=float32), array([-0.5, -0.5, -0.5, -0.5], dtype=float32)]\n", "\n", "unit = minibatch\n", " [array([[-0.25, -0.25],\n", " [-0.25, -0.25],\n", " [-0.25, -0.25],\n", " [-0.25, -0.25]], dtype=float32), array([-0.25, -0.25], dtype=float32), array([[-0.25, -0.25, -0.25, -0.25],\n", " [-0.25, -0.25, -0.25, -0.25],\n", " [-0.25, -0.25, -0.25, -0.25]], dtype=float32), array([-0.25, -0.25, -0.25, -0.25], dtype=float32)]\n" ] } ], "source": [ "def inspect_update(learner, mbsize, count=1):\n", " # Save current parameter values\n", " old_values = [p.value for p in learner.parameters]\n", " # Set current parameter values to all 0\n", " for p in learner.parameters:\n", " p.value = 0 * p.value\n", " # create all 1 gradients and associate them with the parameters\n", " updates = {p: p.value + 1 for p in learner.parameters} \n", " # do 'count' many updates\n", " for i in range(count):\n", " learner.update(updates, mbsize)\n", " ret_values = [p.value for p in learner.parameters]\n", " # Restore values\n", " for p, o in zip(learner.parameters, old_values):\n", " p.value = o\n", " return ret_values\n", "\n", "print('\\nunit = sample\\n', inspect_update(sgd_learner_s, mbsize=2))\n", "print('\\nunit = minibatch\\n', inspect_update(sgd_learner_m, mbsize=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With the knowledge that SGD is the update `parameter = old_parameter - learning_rate * gradient`, we can conclude that when the learning rate schedule is per minibatch, the learning rate is divided by the minibatch size, when the learning rate schedule is per sample the learning rate is not divided by the minibatch size. CNTK offers both options because in some setups it is more convenient to work with per sample learning rates than per minibatch learning rates and vice versa. For example, per minibatch learning rate schedules, typically don't require retuning when you want to change the minibatch size, but per sample schedules do. On the other hand with distributed training it is more correct to specify the learning rate schedule as per sample rather than per minibatch.\n", "\n", "Calling update manually on the learner (as `inspect_update` does) is very tedious. Besides, you need to arrange for the gradients to be computed and provided to the learner. Fortunately, if you use a **`Trainer`**, you don't have to do any of that. \n", "Outside this document you will never see any examples manually calling update on the learner! \n", "\n", "## Trainers and Learners\n", "\n", "A closely related class to the `Learner` is the `Trainer`. In CNTK a `Trainer` brings together all the ingredients necessary for training models:\n", "- the model itself\n", "- the loss function (a differentiable function) and the evaluation function which is not necessarily differentiable (such as error rate)\n", "- the learners\n", "- optionally progress writers that log the training progress\n", "\n", "While in the most typical case a `Trainer` has a single learner that handles all the parameters, it is possible to have **multiple learners** each working on a different subset of the parameters. Parameters that are not covered by any learner **will not** be updated. Here is an example that illustrates typical use.\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false }, "outputs": [], "source": [ "lr_schedule = C.learning_rate_schedule([0.05]*3 + [0.025]*2 + [0.0125], C.UnitType.minibatch, epoch_size=100)\n", "sgd_learner = C.sgd(z.parameters, lr_schedule)\n", "loss = C.cross_entropy_with_softmax(z, label)\n", "trainer = C.Trainer(z, loss, sgd_learner)\n", "# use the trainer with a minibatch source as in the trainer howto" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The trainer will compute the gradients of `loss` with respect to the parameters of `z` and call the sgd_learner's update method as we did manually in the `inspect_update` function earlier. Here we have specified a learning rate schedule that is 0.05 for the first 300 minibatches (3 times the epoch size), then drops to 0.025 for the next 200 minibatches, and it is 0.0125 from then on until the end of training. This kind of functionality is quite common in tuning neural networks and it is the reason why in some papers (such as the [ResNet paper](https://arxiv.org/abs/1512.03385)) we see learning curves like this\n", "![resnet](resnet.png)\n", "\n", "What is happening here is that the learning rate gets reduced by a factor of 10 after 150000 and 300000 updates (cf. section 3.4 of the paper)\n", "\n", "Finally the `cntk.Function.train` method provides convenience functionality which allows you to specify the learner and the data and it internally creates a trainer that drives the training loop.\n", "\n", "## Other builtin learners\n", "\n", "Apart from SGD, other builtin learners include \n", "- SGD with momentum (`momentum_sgd()`)\n", "- SGD with Nesterov momentum (`nesterov()`) first popularized in deep learning by [this paper](http://proceedings.mlr.press/v28/sutskever13.html)\n", "- Adagrad (`adagrad()`) first popularized in deep learning by [this paper](https://research.google.com/archive/large_deep_networks_nips2012.html) \n", "- RMSProp (`rmsprop()`) a correction to adagrad that prevents the learning rate from decaying too fast.\n", "- FSAdagrad (`fsadagrad()`) adds momentum and bias correction to RMSprop\n", "- Adam / Adamax (`adam(..., adamax=False/True)`) see [this paper](https://arxiv.org/abs/1412.6980)\n", "- Adadelta (`adadelta()`) see [this paper](https://arxiv.org/abs/1212.5701)\n", "\n", "### Momentum\n", "\n", "Among these learners, `momentum_sgd`, `nesterov`, `fsadagrad`, and `adam` take an additional momemtum schedule. \n", "\n", "Momentum means that instead of updating the parameter using the current gradient we update the parameter using all previous gradients exponentially decayed . If there is a consistent direction that the gradients are pointing to, the parameter updates will develop momentum in that direction. [This page](http://distill.pub/2017/momentum/) has a good explanation of momentum.\n", "\n", "Like the learning rate schedule, the momentum schedule can be specified in two equivalent ways:\n", "- `momentum_schedule(float or list of floats, epoch_size)`\n", "- `momentum_as_time_constant(float or list of floats, epoch_size)`\n", "\n", "As with `learning_rate_schedule` the arguments are interpreted in the same way, i.e. there's flexibility is specifying different momentum for the first few minibatches and for later minibatches. \n", "\n", "The difference between the two calls is just a simple transformation which can be explained as follows. Since momentum is creating a sort of exponential moving average it is fair to ask \"when does the contribution of an old gradient diminish by a certain constant factor?\". If we choose the constant factor to be $0.5$ we call this the [half-life](https://en.wikipedia.org/wiki/Half-life) and if we choose the constant to be $e^{-1}\\approx 0.368$ we call this the [time constant](https://en.wikipedia.org/wiki/Time_constant). So `momentum_as_time_constant_schedule` specifies the number of samples it would take for a gradient to decay to $0.368$ of its original contribution on the momentum term. Specifying a `momentum_as_time_constant_schedule(300)` is a little bit more meaningful than specifying `momentum_schedule(.967...)` even though both lead to the same updates. The way to convert between the two schedules is\n", "- $\\textrm{momentum} = \\exp(-\\frac{\\textrm{minibatch_size}}{\\textrm{time_constant}})$\n", "- $\\textrm{time_constant} = \\frac{\\textrm{minibatch_size}}{\\log(1/\\textrm{momentum})}$\n", "\n", "Apart from the momentum schedule, the momentum learners can also take a boolean \"unit_gain\" argument that determines the form of the momentum update:\n", "- `unit_gain=True`: $\\textrm{momentum_direction} = \\textrm{momentum} \\cdot \\textrm{old_momentum_direction} + (1 - \\textrm{momentum}) \\cdot \\textrm{gradient}$\n", "- `unit_gain=False`: $\\textrm{momentum_direction} = \\textrm{momentum} \\cdot \\textrm{old_momentum_direction} + \\textrm{gradient}$\n", "\n", "The idea behind the non-conventional `unit_gain=True` is that when momentum and or learning rate changes this way of updating does not lead to divergence.\n", "\n", "The following code illustrates that for the case of `unit_gain=False`, the two ways of specifying momentum (as time constant or not) are equivalent. It also shows that when `unit_gain=True` you need to scale your learning rate by $1/(1-\\textrm{momentum})$ to match the `unit_gain=False` case" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "time constant for momentum of 0.967... = 300.00000000000006\n", "momentum for time constant of 300 = 0.9672161004820059\n", "[-1.436 -1.436]\n", "[-1.436 -1.436]\n", "[-1.436 -1.436]\n" ] } ], "source": [ "mb_size = 10\n", "time_constant = 300\n", "momentum = math.exp(-mb_size/time_constant)\n", "\n", "print('time constant for momentum of 0.967... = ', mb_size/math.log(1/momentum))\n", "print('momentum for time constant of 300 = ', math.exp(-mb_size/time_constant))\n", "\n", "lr_schedule = C.learning_rate_schedule(1, C.UnitType.minibatch)\n", "ug_schedule = C.learning_rate_schedule(1/(1-momentum), C.UnitType.minibatch)\n", "\n", "m_schedule = C.momentum_schedule(momentum)\n", "t_schedule = C.momentum_as_time_constant_schedule(time_constant)\n", "\n", "msgd = C.momentum_sgd(z.parameters, lr_schedule, m_schedule, unit_gain=False)\n", "tsgd = C.momentum_sgd(z.parameters, lr_schedule, t_schedule, unit_gain=False)\n", "usgd = C.momentum_sgd(z.parameters, ug_schedule, m_schedule, unit_gain=True)\n", "\n", "print(inspect_update(msgd, mb_size, 5)[0][0])\n", "print(inspect_update(tsgd, mb_size, 5)[0][0])\n", "print(inspect_update(usgd, mb_size, 5)[0][0])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Learners with individual learning rates\n", "\n", "Among the builtin learners, `adagrad`, `rmsprop`, `fsadagrad`, `adam`, and `adadelta` have rules for tuning the learning rate of each parameter individually. They still require the tuning of a global learning rate that gets multiplied with the individual learning rate of each parameter but the hope is that these techniques can achieve better results by taking advantage of the fact that, for example, if the inputs are word ids then some features appear more rarely than others and therefore the learning rate for the parameters associated with rare feature should depend on how often those features have been seen rather than how many minibatches have been processed. \n", "\n", "These methods are typically easier to tune, but there is some new evidence that [they overfit more easily](https://arxiv.org/abs/1705.08292) than SGD with momentum.\n", "\n", "Below we show how these learners can be configured and the effect their updates have on the parameters. \n", "The main take-away is that **if you switch learners, you need to retune the learning rate**. In this example the sequence of initial points and gradients is the same yet different learners arrive at different parameter values after 10 minibatches. However if we retune the learning rates, the learner with the smallest parameter value (adadelta), we can reach as large parameter values as the one with the largest parameter value (adamax). Also this is an artificial example where gradients are consistently equal to 1 so the methods that have momemtum builtin (`adam`/`adamax`/`fsadagrad`) should get higher parameter values than the methods that don't have momentum builtin (for the same value of the learning rate)." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "adadelta : [-0.0099 -0.0099]\n", "agagrad : [-0.3125 -0.3125]\n", "adam : [-9.9203 -9.9203]\n", "adamax : [-9.9227 -9.9227]\n", "fsadagrad: [-8.8573 -8.8573]\n", "rmsprop : [-0.3125 -0.3125]\n", "adadelta2: [-9.9228 -9.9228]\n" ] } ], "source": [ "mb_size = 32\n", "time_constant = 1000\n", "\n", "lr_schedule = C.learning_rate_schedule(1, C.UnitType.minibatch)\n", "t_schedule = C.momentum_as_time_constant_schedule(time_constant)\n", "\n", "tsgd = C.momentum_sgd(z.parameters, lr_schedule, t_schedule, unit_gain=False)\n", "\n", "adadelta = C.adadelta(z.parameters, lr_schedule, 0.999, 1e-6)\n", "adagrad = C.adagrad(z.parameters, lr_schedule)\n", "adam = C.adam(z.parameters, lr_schedule, t_schedule, unit_gain=False)\n", "adamax = C.adam(z.parameters, lr_schedule, t_schedule, unit_gain=False, adamax=True)\n", "fsadagrad = C.fsadagrad(z.parameters, lr_schedule, t_schedule, unit_gain=False)\n", "rmsprop = C.rmsprop(z.parameters, lr_schedule, gamma=0.999, inc=1.0+1e-9, dec=1.0-1e-9, max=np.inf, min=1e-30)\n", "\n", "print('adadelta :', inspect_update(adadelta, mb_size, 10)[0][0])\n", "print('agagrad :', inspect_update(adagrad, mb_size, 10)[0][0])\n", "print('adam :', inspect_update(adam, mb_size, 10)[0][0])\n", "print('adamax :', inspect_update(adamax, mb_size, 10)[0][0])\n", "print('fsadagrad:', inspect_update(fsadagrad, mb_size, 10)[0][0])\n", "print('rmsprop :', inspect_update(rmsprop, mb_size, 10)[0][0])\n", "\n", "adadelta_schedule = C.learning_rate_schedule(1004, C.UnitType.minibatch)\n", "adadelta_tuned = C.adadelta(z.parameters, adadelta_schedule, 0.999, 1e-6)\n", "print('adadelta2:', inspect_update(adadelta_tuned, mb_size, 10)[0][0])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Writing a learner as a CNTK expression\n", "\n", "If you want to experiment with your own learner, you should first try to write it as a CNTK expression. This is much faster than the next alternative, which is to write it in Python. CNTK has a universal learner that accepts a function as an argument. This function takes a list of parameters and gradients and creates an expression (a network) that when evaluated it will assign new values to the parameters according to the learning rule you coded. At the time of this writing, this learner does not support schedules for learning rate and momentum. If this is necessary, the user must create a new learner. Another deficiency of this learner is it only supports densely stored gradients. If you get an error that a quantity is not dense, you have two options:\n", "- find the parameters with sparse gradients (typically those used at the very first layer) and put them in a builtin learner\n", "- replace input variables that are sparse with dense (is_sparse=False)\n", "\n", "Below we show how to write RMSprop using the universal learner." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[-0.5397 -0.5397]\n" ] } ], "source": [ "def my_rmsprop(parameters, gradients):\n", " rho = 0.999\n", " lr = 0.01\n", " # We use the following accumulator to store the moving average of every squared gradient\n", " accumulators = [C.constant(1e-6, shape=p.shape, dtype=p.dtype) for p in parameters]\n", " update_funcs = []\n", " for p, g, a in zip(parameters, gradients, accumulators):\n", " # We declare that `a` will be replaced by an exponential moving average of squared gradients\n", " # The return value is the expression rho * a + (1-rho) * g * g \n", " accum_new = C.assign(a, rho * a + (1-rho) * g * g)\n", " # This is the rmsprop update. \n", " # We need to use accum_new to create a dependency on the assign statement above. \n", " # This way, when we run this network both assigns happen.\n", " update_funcs.append(C.assign(p, p - lr * g / C.sqrt(accum_new)))\n", " return C.combine(update_funcs)\n", "\n", "my_learner = C.universal(my_rmsprop, z.parameters)\n", "print(inspect_update(my_learner, 10, 2)[0][0])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Writing a learner as a Python class\n", "\n", "CNTK expressions are very powerful and all the well-known learners can be expressed in this way. Still, there can be rare cases where you want to perform an update that cannot be currently implemented as a CNTK expression. In those cases you can implement your learner as a Python class. CNTK will then call its update method during training. Since this means C++ (the training loop) is calling into Python (your learner) for every single minibatch, this approach is the slowest of all options.\n", "\n", "In order for your class to be understood as a learner it has to inherit from `cntk.UserLearner`. The constructor can be used to set up the learner and the trainer will call the learner's update method by supplying it a dictionary whose keys are the parameters and whose values are the corresponding gradients, as well as the number of samples in the minibatch and whether we have reached the end of a sweep through the data. The implementation of update is totally up to you.\n", "\n", "In the code below, we create a learner that just performs SGD. In the constructor we create a dictionary mapping tensor shapes to CNTK expressions with the gradients being input variables. In the update method for each parameter, gradient pair we look up the expression corresponding to the shape of the parameter, bind the gradient to the input of the expression and evaluate the expression. Finally we slice the result to get rid of the batch axis and update the parameter. We have also slightly modified the `inspect_update` method to make it work with a user defined learner." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[-0.1562 -0.1562]\n" ] } ], "source": [ "class MySgd(C.UserLearner):\n", "\n", " def __init__(self, parameters, lr_schedule):\n", " super(MySgd, self).__init__(parameters, lr_schedule, as_numpy=False)\n", "\n", " self.new_p = {}\n", " self.grad_input = {}\n", "\n", " self.sample_count_input = C.input_variable((), name='count')\n", "\n", " lr = lr_schedule[0] # assuming constant learning rate\n", " eta = lr / self.sample_count_input\n", "\n", " # we need one graph per parameter shape\n", " for param in parameters:\n", " p_shape = param.shape\n", " self.grad_input[p_shape] = C.input_variable(p_shape)\n", " self.new_p[p_shape] = param - eta * self.grad_input[p_shape]\n", "\n", " def update(self, gradient_values, training_sample_count, sweep_end):\n", " for p, g in gradient_values.items():\n", " new_p = self.new_p[p.shape]\n", " grad_input = self.grad_input[p.shape]\n", "\n", " data = {\n", " self.sample_count_input: np.asarray(training_sample_count),\n", " grad_input: g\n", " }\n", " result = new_p.eval(data, as_numpy=False)\n", " shape = result.shape\n", "\n", " # result has the shape of a complete minibatch, but contains\n", " # only one tensor, which we want to write to p. This means, we\n", " # have to slice off the leading dynamic axis.\n", " static_tensor = result.data.slice_view([0]*len(shape), shape[1:])\n", " p.set_value(static_tensor)\n", " return True\n", " \n", "mb_size = 64\n", "lr_schedule = C.learning_rate_schedule(1, C.UnitType.minibatch)\n", "my_sgd = MySgd(z.parameters, lr_schedule)\n", "\n", "def inspect_user_learner_update(learner, mbsize, count):\n", " easy_parameters = [C.Variable.as_parameter(p) for p in learner.parameters()]\n", " old_values = [p.value for p in easy_parameters]\n", " for p in easy_parameters:\n", " p.value = 0 * p.value\n", " updates = {p: p.value + 1 for p in easy_parameters}\n", " for i in range(count):\n", " learner.update(updates, np.float32(mbsize), sweep_end=False)\n", " ret_values = [p.value for p in easy_parameters]\n", " for p, o in zip(easy_parameters, old_values):\n", " p.value = o\n", " return ret_values\n", "\n", "print(inspect_user_learner_update(my_sgd, mb_size, 10)[0][0])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And that's all there is to learners! They are at the heart of neural network training, but by themselves they are not very useful, and they are typically driven by a trainer. So a good next step for you would be to take a look at our Trainer howto." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "Python [default]", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "nbformat_minor": 1 }