Revision 1db48f3a735eb0fba06a7d503f080a7ead512604 authored by Artem Artemev on 11 July 2018, 12:50:44 UTC, committed by GitHub on 11 July 2018, 12:50:44 UTC
1 parent 707b195
monitor-tensorboard.ipynb
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import itertools\n",
"import os\n",
"os.environ['CUDA_VISIBLE_DEVICES']=\"\"\n",
"import numpy as np\n",
"import gpflow\n",
"import gpflow.training.monitor as mon\n",
"import numbers\n",
"import matplotlib.pyplot as plt\n",
"import tensorflow as tf"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Demo: `gpflow.training.monitor`\n",
"In this notebook we'll demo how to use `gpflow.training.monitor` for logging the optimisation of a GPflow model.\n",
"\n",
"## Creating the GPflow model\n",
"We first generate some random data and create a GPflow model.\n",
"\n",
"Under the hood, GPflow gives a unique name to each model which is used to name the Variables it creates in the TensorFlow graph containing a random identifier. This is useful in interactive sessions, where people may create a few models, to prevent variables with the same name conflicting. However, when loading the model, we need to make sure that the names of all the variables are exactly the same as in the checkpoint. This is why we pass name=\"SVGP\" to the model constructor, and why we use gpflow.defer_build()."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"np.random.seed(0)\n",
"X = np.random.rand(10000, 1) * 10\n",
"Y = np.sin(X) + np.random.randn(*X.shape)\n",
"Xt = np.random.rand(10000, 1) * 10\n",
"Yt = np.sin(Xt) + np.random.randn(*Xt.shape)\n",
"\n",
"with gpflow.defer_build():\n",
" m = gpflow.models.SVGP(X, Y, gpflow.kernels.RBF(1), gpflow.likelihoods.Gaussian(),\n",
" Z=np.linspace(0, 10, 5)[:, None],\n",
" minibatch_size=100, name=\"SVGP\")\n",
" m.likelihood.variance = 0.01\n",
"m.compile()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's compute log likelihood before the optimisation"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"LML before the optimisation: -1271605.621944\n"
]
}
],
"source": [
"print('LML before the optimisation: %f' % m.compute_log_likelihood())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will be using a TensorFlow optimiser. All TensorFlow optimisers have a support for `global_step` variable. Its purpose is to track how many optimisation steps have occurred. It is useful to keep this in a TensorFlow variable as this allows it to be restored together with all the parameters of the model.\n",
"\n",
"The code below creates this variable using a monitor's helper function. It is important to create it before building the monitor in case the monitor includes a checkpoint task. This is because the checkpoint internally uses the TensorFlow Saver which creates a list of variables to save. Therefore all variables expected to be saved by the checkpoint task should exist by the time the task is created."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"session = m.enquire_session()\n",
"global_step = mon.create_global_step(session)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Construct the monitor\n",
"\n",
"Next we need to construct the monitor. `gpflow.training.monitor` provides classes that are building blocks for the monitor. Essengially, a monitor is a function that is provided as a callback to an optimiser. It consists of a number of tasks that may be executed at each step, subject to their running condition.\n",
"\n",
"In this example, we want to:\n",
"- log certain scalar parameters in TensorBoard,\n",
"- log the full optimisation objective (log marginal likelihood bound) periodically, even though we optimise with minibatches,\n",
"- store a backup of the optimisation process periodically,\n",
"- log performance for a test set periodically.\n",
"\n",
"We will define these tasks as follows:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"print_task = mon.PrintTimingsTask().with_name('print')\\\n",
" .with_condition(mon.PeriodicIterationCondition(10))\\\n",
" .with_exit_condition(True)\n",
"\n",
"sleep_task = mon.SleepTask(0.01).with_name('sleep').with_name('sleep')\n",
"\n",
"saver_task = mon.CheckpointTask('./monitor-saves').with_name('saver')\\\n",
" .with_condition(mon.PeriodicIterationCondition(10))\\\n",
" .with_exit_condition(True)\n",
"\n",
"file_writer = mon.LogdirWriter('./model-tensorboard')\n",
"\n",
"model_tboard_task = mon.ModelToTensorBoardTask(file_writer, m).with_name('model_tboard')\\\n",
" .with_condition(mon.PeriodicIterationCondition(10))\\\n",
" .with_exit_condition(True)\n",
"\n",
"lml_tboard_task = mon.LmlToTensorBoardTask(file_writer, m).with_name('lml_tboard')\\\n",
" .with_condition(mon.PeriodicIterationCondition(100))\\\n",
" .with_exit_condition(True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As the above code shows, each task can be assigned a name and running conditions. The name will be shown in the task timing summary.\n",
"\n",
"There are two different types of running conditions: `with_condition` controls execution of the task at each iteration in the optimisation loop. `with_exit_condition` is a simple boolean flag indicating that the task should also run at the end of optimisation.\n",
"In this example we want to run our tasks periodically, at every iteration or every 10th or 100th iteration.\n",
"\n",
"Notice that the two TensorBoard tasks will write events into the same file. It is possible to share a file writer between multiple tasks. However it is not possible to share the same event location between multiple file writers. An attempt to open two writers with the same location will result in error.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Custom tasks\n",
"We may also want to perfom certain tasks that do not have pre-defined `Task` classes. For example, we may want to compute the performance on a test set. Here we create such a class by extending `BaseTensorBoardTask` to log the testing benchmarks in addition to all the scalar parameters."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"class CustomTensorBoardTask(mon.BaseTensorBoardTask):\n",
" def __init__(self, file_writer, model, Xt, Yt):\n",
" super().__init__(file_writer, model)\n",
" self.Xt = Xt\n",
" self.Yt = Yt\n",
" self._full_test_err = tf.placeholder(gpflow.settings.tf_float, shape=())\n",
" self._full_test_nlpp = tf.placeholder(gpflow.settings.tf_float, shape=())\n",
" self._summary = tf.summary.merge([tf.summary.scalar(\"test_rmse\", self._full_test_err),\n",
" tf.summary.scalar(\"test_nlpp\", self._full_test_nlpp)])\n",
" \n",
" def run(self, context: mon.MonitorContext, *args, **kwargs) -> None:\n",
" minibatch_size = 100\n",
" preds = np.vstack([self.model.predict_y(Xt[mb * minibatch_size:(mb + 1) * minibatch_size, :])[0]\n",
" for mb in range(-(-len(Xt) // minibatch_size))])\n",
" test_err = np.mean((Yt - preds) ** 2.0)**0.5\n",
" self._eval_summary(context, {self._full_test_err: test_err, self._full_test_nlpp: 0.0})\n",
"\n",
" \n",
"custom_tboard_task = CustomTensorBoardTask(file_writer, m, Xt, Yt).with_name('custom_tboard')\\\n",
" .with_condition(mon.PeriodicIterationCondition(100))\\\n",
" .with_exit_condition(True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can put all these tasks into a monitor."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"monitor_tasks = [print_task, model_tboard_task, lml_tboard_task, custom_tboard_task, saver_task, sleep_task]\n",
"monitor = mon.Monitor(monitor_tasks, session, global_step)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Running the optimisation\n",
"We finally get to running the optimisation.\n",
"\n",
"We may want to continue a previously run optimisation by resotring the TensorFlow graph from the latest checkpoint. Otherwise skip this step."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"if os.path.isdir('./monitor-saves'):\n",
" mon.restore_session(session, './monitor-saves')"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"scrolled": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Iteration 10\ttotal itr.rate 12.98/s\trecent itr.rate 12.98/s\topt.step 10\ttotal opt.rate 14.72/s\trecent opt.rate 14.72/s\n",
"Iteration 20\ttotal itr.rate 19.32/s\trecent itr.rate 37.77/s\topt.step 20\ttotal opt.rate 28.96/s\trecent opt.rate 887.41/s\n",
"Iteration 30\ttotal itr.rate 24.44/s\trecent itr.rate 51.97/s\topt.step 30\ttotal opt.rate 42.54/s\trecent opt.rate 690.68/s\n",
"Iteration 40\ttotal itr.rate 27.92/s\trecent itr.rate 48.73/s\topt.step 40\ttotal opt.rate 55.70/s\trecent opt.rate 771.83/s\n",
"Iteration 50\ttotal itr.rate 30.70/s\trecent itr.rate 51.09/s\topt.step 50\ttotal opt.rate 68.73/s\trecent opt.rate 1068.65/s\n",
"Iteration 60\ttotal itr.rate 33.03/s\trecent itr.rate 53.21/s\topt.step 60\ttotal opt.rate 80.79/s\trecent opt.rate 658.99/s\n",
"Iteration 70\ttotal itr.rate 34.66/s\trecent itr.rate 49.25/s\topt.step 70\ttotal opt.rate 92.57/s\trecent opt.rate 741.85/s\n",
"Iteration 80\ttotal itr.rate 35.12/s\trecent itr.rate 38.67/s\topt.step 80\ttotal opt.rate 103.80/s\trecent opt.rate 687.96/s\n",
"Iteration 90\ttotal itr.rate 36.37/s\trecent itr.rate 50.92/s\topt.step 90\ttotal opt.rate 114.64/s\trecent opt.rate 696.17/s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 27%|██▋ | 27/100 [00:00<00:00, 268.51it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Iteration 100\ttotal itr.rate 37.33/s\trecent itr.rate 48.98/s\topt.step 100\ttotal opt.rate 125.23/s\trecent opt.rate 743.73/s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 100/100 [00:00<00:00, 391.92it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Iteration 110\ttotal itr.rate 31.69/s\trecent itr.rate 12.62/s\topt.step 110\ttotal opt.rate 135.53/s\trecent opt.rate 760.34/s\n",
"Iteration 120\ttotal itr.rate 32.47/s\trecent itr.rate 44.38/s\topt.step 120\ttotal opt.rate 144.96/s\trecent opt.rate 618.62/s\n",
"Iteration 130\ttotal itr.rate 33.50/s\trecent itr.rate 54.19/s\topt.step 130\ttotal opt.rate 155.35/s\trecent opt.rate 1114.54/s\n",
"Iteration 140\ttotal itr.rate 34.41/s\trecent itr.rate 53.25/s\topt.step 140\ttotal opt.rate 165.32/s\trecent opt.rate 995.97/s\n",
"Iteration 150\ttotal itr.rate 35.22/s\trecent itr.rate 52.35/s\topt.step 150\ttotal opt.rate 174.51/s\trecent opt.rate 787.66/s\n",
"Iteration 160\ttotal itr.rate 35.73/s\trecent itr.rate 45.79/s\topt.step 160\ttotal opt.rate 183.04/s\trecent opt.rate 684.73/s\n",
"Iteration 170\ttotal itr.rate 36.26/s\trecent itr.rate 47.44/s\topt.step 170\ttotal opt.rate 191.39/s\trecent opt.rate 708.79/s\n",
"Iteration 180\ttotal itr.rate 36.82/s\trecent itr.rate 50.08/s\topt.step 180\ttotal opt.rate 199.34/s\trecent opt.rate 677.99/s\n",
"Iteration 190\ttotal itr.rate 37.30/s\trecent itr.rate 48.74/s\topt.step 190\ttotal opt.rate 207.55/s\trecent opt.rate 803.13/s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 35%|███▌ | 35/100 [00:00<00:00, 343.13it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Iteration 200\ttotal itr.rate 37.75/s\trecent itr.rate 48.76/s\topt.step 200\ttotal opt.rate 215.43/s\trecent opt.rate 774.15/s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 100/100 [00:00<00:00, 419.77it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Iteration 210\ttotal itr.rate 35.44/s\trecent itr.rate 15.96/s\topt.step 210\ttotal opt.rate 222.91/s\trecent opt.rate 728.65/s\n",
"Iteration 220\ttotal itr.rate 35.90/s\trecent itr.rate 49.17/s\topt.step 220\ttotal opt.rate 231.35/s\trecent opt.rate 1128.91/s\n",
"Iteration 230\ttotal itr.rate 36.32/s\trecent itr.rate 48.79/s\topt.step 230\ttotal opt.rate 238.52/s\trecent opt.rate 748.70/s\n",
"Iteration 240\ttotal itr.rate 36.60/s\trecent itr.rate 44.57/s\topt.step 240\ttotal opt.rate 245.11/s\trecent opt.rate 673.52/s\n",
"Iteration 250\ttotal itr.rate 36.96/s\trecent itr.rate 48.38/s\topt.step 250\ttotal opt.rate 249.71/s\trecent opt.rate 454.11/s\n",
"Iteration 260\ttotal itr.rate 37.28/s\trecent itr.rate 47.77/s\topt.step 260\ttotal opt.rate 256.17/s\trecent opt.rate 726.32/s\n",
"Iteration 270\ttotal itr.rate 37.60/s\trecent itr.rate 48.28/s\topt.step 270\ttotal opt.rate 262.60/s\trecent opt.rate 754.52/s\n",
"Iteration 280\ttotal itr.rate 37.88/s\trecent itr.rate 47.34/s\topt.step 280\ttotal opt.rate 268.87/s\trecent opt.rate 757.42/s\n",
"Iteration 290\ttotal itr.rate 38.16/s\trecent itr.rate 48.09/s\topt.step 290\ttotal opt.rate 274.98/s\trecent opt.rate 755.01/s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 31%|███ | 31/100 [00:00<00:00, 308.24it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Iteration 300\ttotal itr.rate 38.42/s\trecent itr.rate 47.92/s\topt.step 300\ttotal opt.rate 280.92/s\trecent opt.rate 752.18/s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 100/100 [00:00<00:00, 406.33it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Iteration 310\ttotal itr.rate 36.72/s\trecent itr.rate 15.79/s\topt.step 310\ttotal opt.rate 286.54/s\trecent opt.rate 717.99/s\n",
"Iteration 320\ttotal itr.rate 36.98/s\trecent itr.rate 47.48/s\topt.step 320\ttotal opt.rate 292.08/s\trecent opt.rate 728.84/s\n",
"Iteration 330\ttotal itr.rate 37.27/s\trecent itr.rate 49.85/s\topt.step 330\ttotal opt.rate 297.34/s\trecent opt.rate 701.64/s\n",
"Iteration 340\ttotal itr.rate 37.51/s\trecent itr.rate 47.65/s\topt.step 340\ttotal opt.rate 302.74/s\trecent opt.rate 756.13/s\n",
"Iteration 350\ttotal itr.rate 37.72/s\trecent itr.rate 46.16/s\topt.step 350\ttotal opt.rate 305.89/s\trecent opt.rate 472.85/s\n",
"Iteration 360\ttotal itr.rate 37.99/s\trecent itr.rate 51.26/s\topt.step 360\ttotal opt.rate 311.03/s\trecent opt.rate 755.88/s\n",
"Iteration 370\ttotal itr.rate 38.20/s\trecent itr.rate 47.40/s\topt.step 370\ttotal opt.rate 316.00/s\trecent opt.rate 742.50/s\n",
"Iteration 380\ttotal itr.rate 38.39/s\trecent itr.rate 47.28/s\topt.step 380\ttotal opt.rate 320.64/s\trecent opt.rate 702.43/s\n",
"Iteration 390\ttotal itr.rate 38.62/s\trecent itr.rate 50.03/s\topt.step 390\ttotal opt.rate 325.46/s\trecent opt.rate 760.64/s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 32%|███▏ | 32/100 [00:00<00:00, 316.34it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Iteration 400\ttotal itr.rate 38.82/s\trecent itr.rate 48.42/s\topt.step 400\ttotal opt.rate 330.29/s\trecent opt.rate 783.85/s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 100/100 [00:00<00:00, 403.00it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Iteration 410\ttotal itr.rate 37.50/s\trecent itr.rate 15.92/s\topt.step 410\ttotal opt.rate 334.72/s\trecent opt.rate 721.82/s\n",
"Iteration 420\ttotal itr.rate 37.69/s\trecent itr.rate 47.42/s\topt.step 420\ttotal opt.rate 339.00/s\trecent opt.rate 711.56/s\n",
"Iteration 430\ttotal itr.rate 37.90/s\trecent itr.rate 49.11/s\topt.step 430\ttotal opt.rate 343.45/s\trecent opt.rate 765.62/s\n",
"Iteration 440\ttotal itr.rate 38.08/s\trecent itr.rate 48.17/s\topt.step 440\ttotal opt.rate 347.65/s\trecent opt.rate 734.16/s\n",
"Iteration 450\ttotal itr.rate 38.26/s\trecent itr.rate 48.47/s\topt.step 450\ttotal opt.rate 351.91/s\trecent opt.rate 763.46/s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 49%|████▉ | 49/100 [00:00<00:00, 482.62it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Iteration 450\ttotal itr.rate 37.49/s\trecent itr.rate 0.00/s\topt.step 450\ttotal opt.rate 317.80/s\trecent opt.rate 0.00/s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 100/100 [00:00<00:00, 483.37it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Tasks execution time summary:\n",
"print:\t0.0373 (sec)\n",
"model_tboard:\t0.1832 (sec)\n",
"lml_tboard:\t1.2192 (sec)\n",
"custom_tboard:\t1.1748 (sec)\n",
"saver:\t3.8945 (sec)\n",
"sleep:\t4.5394 (sec)\n"
]
}
],
"source": [
"optimiser = gpflow.train.AdamOptimizer(0.01)\n",
"\n",
"with mon.Monitor(monitor_tasks, session, global_step, print_summary=True) as monitor:\n",
" optimiser.minimize(m, step_callback=monitor, maxiter=450, global_step=global_step)\n",
"\n",
"file_writer.close()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now lets compute the log likelihood again. Hopefully we will see an increase in its value"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"LML after the optimisation: -68705.124191\n"
]
}
],
"source": [
"print('LML after the optimisation: %f' % m.compute_log_likelihood())"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python GPFlow-venv",
"language": "python",
"name": "gpflow_venv"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Computing file changes ...
