CMLnn.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "CMLnn.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jn7TH8rUIYM3",
        "colab_type": "text"
      },
      "source": [
        "![tent map CMLs](https://media-exp1.licdn.com/dms/image/C4D22AQF5qqMBVPHWHg/feedshare-shrink_1280/0?e=1593043200&v=beta&t=26DxI5bLMUm3eEH9vbRyelF5Z79XAkJwWA9BWdC_3zQ)\n",
        "# Learning Chaotic Coupled-Map Lattices\n",
        "Two main goals:\n",
        "- learning to initialize chaotic systems\n",
        "- learning to configure chaotic systems (both ICs and BPs)\n",
        "\n",
        "The first seems to be much easier than the second.\n",
        "\n",
        "## Todos\n",
        "- address vanishing gradients for larger timesteps\n",
        "  - residual connection doesn't seem to solve the problem...\n",
        "  - could also try a different map function (tanh?)\n",
        "  - [Shobu-Ose-Mori](http://www.scipress.org/journals/forma/pdf/2801/28010001.pdf) Map is piecewise liear, has stronger grads\n",
        "  - tent map (would need to be made diffable at center)\n",
        "  - [smooth generalization](https://www.worldscientific.com/doi/10.1142/S021812741730004X) of tent/logistic\n",
        "  - [circle map](https://manasataramgini.wordpress.com/2017/11/26/pattern-formation-in-coupled-map-lattices-with-the-circle-map-tanh-map-and-chebyshev-map/)\n",
        "\n",
        "## Errata\n",
        "- [role of ICs](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4934879/) in pattern-selection dynamics "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NiuxlX-HUweH",
        "colab_type": "text"
      },
      "source": [
        "### Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UgUE_cJvUx3F",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from __future__ import absolute_import, division, print_function, unicode_literals\n",
        "# To generate GIFs\n",
        "!pip install -q imageio\n",
        "import glob\n",
        "import imageio\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import os\n",
        "import PIL\n",
        "import time\n",
        "from tqdm import tqdm\n",
        "import tensorflow as tf\n",
        "from tensorflow.keras.callbacks import TensorBoard\n",
        "import sys\n",
        "import cv2\n",
        "\n",
        "from IPython import display\n",
        "\n",
        "K = tf.keras.backend\n",
        "layers = tf.keras.layers\n",
        "\n",
        "from tensorflow.keras.datasets import mnist"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YGYHL7lzXqya",
        "colab_type": "text"
      },
      "source": [
        "## Constants / Training Parameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mt73CoCNXr2a",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "batch_size = 512\n",
        "BATCH_SIZE = batch_size\n",
        "num_classes = 10\n",
        "epochs = 100\n",
        "EPOCHS = epochs\n",
        "\n",
        "dynamics = {'chaotic traveling wave': (1.69, 0.5),\n",
        "            'spatiotemporal intermittency': (1.7522, 0.00115),\n",
        "            'pattern selection': (1.71, 0.4),\n",
        "            'no coupling': (1.7522, 0.0),\n",
        "            'global coupling': (1.7522, 1.0),\n",
        "            'tent': (0.8, 0.5),\n",
        "            'local tent': (0.8, 0.0),\n",
        "            'zero': (0.0, 0.0)}\n",
        "\n",
        "# input image dimensions\n",
        "img_rows, img_cols = 28, 28\n",
        "\n",
        "TIME=10\n",
        "BP, COUPLING = dynamics['pattern selection']\n",
        "INITIALIZATION_ACTIVATION = \"tanh\" # \"sigmoid\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M8ZweStojRyY",
        "colab_type": "text"
      },
      "source": [
        "## LR Multiplier\n",
        "- to avoid pandemonium, learn system config slower than system initialization (even w/ optimizers)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rv0ExBT_jXRn",
        "colab_type": "code",
        "outputId": "b2d26c46-3605-4928-8541-01a11a7f9c74",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 343
        }
      },
      "source": [
        "!pip install keras-lr-multiplier\n",
        "TF_KERAS = True\n",
        "import keras_lr_multiplier.backend\n",
        "#keras_lr_multiplier.backend.TF_KERAS = True\n",
        "from keras_lr_multiplier import LRMultiplier"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting keras-lr-multiplier\n",
            "  Downloading https://files.pythonhosted.org/packages/7d/78/0eed4862a7274fb491b50881dd2f0dac996ff5774dc4a30c4b628fb78b25/keras-lr-multiplier-0.8.0.tar.gz\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from keras-lr-multiplier) (1.18.4)\n",
            "Requirement already satisfied: Keras in /usr/local/lib/python3.6/dist-packages (from keras-lr-multiplier) (2.3.1)\n",
            "Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-lr-multiplier) (1.0.8)\n",
            "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-lr-multiplier) (1.4.1)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from Keras->keras-lr-multiplier) (3.13)\n",
            "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-lr-multiplier) (1.12.0)\n",
            "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-lr-multiplier) (1.1.2)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from Keras->keras-lr-multiplier) (2.10.0)\n",
            "Building wheels for collected packages: keras-lr-multiplier\n",
            "  Building wheel for keras-lr-multiplier (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras-lr-multiplier: filename=keras_lr_multiplier-0.8.0-cp36-none-any.whl size=5717 sha256=925837f76554f819dcc12e2e1ed638ea03c8f046358c5f05d8962f98d126b7a6\n",
            "  Stored in directory: /root/.cache/pip/wheels/2a/a5/a4/340d5432bced221b2bcca324e3257239784dd1220ab7c786e9\n",
            "Successfully built keras-lr-multiplier\n",
            "Installing collected packages: keras-lr-multiplier\n",
            "Successfully installed keras-lr-multiplier-0.8.0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cIaHglJCIjHC",
        "colab_type": "text"
      },
      "source": [
        "## First test: CMLnet\n",
        "A neural network learns to initialize (and later: bp- and coupling-configure) a coupled map lattice of logistic map units"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UvE19yYSVch6",
        "colab_type": "text"
      },
      "source": [
        "### Logistic Map\n",
        "$x_{n+1} = r \\cdot x_{n}(1 - x_{n})$\n",
        "\n",
        "$x_{n+1} = 1 - ax_{n}^{2}$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Shl2fhT1IU3-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def kaneko_keras(x, a):\n",
        "  #print(\"- Kaneko:\\tx: {}\\ta: {}\".format(x.shape, a.shape))\n",
        "  return K.ones_like(x) - a * K.square(x)\n",
        "\n",
        "def identity(x, a):\n",
        "  return x\n",
        "\n",
        "def tanh(x, a):\n",
        "  return K.tanh(x)\n",
        "\n",
        "# this actually works *better* w/ incorrect domain of [-1, 1]\n",
        "def tent(x, a):\n",
        "  return a * (K.ones_like(x) - 2 * K.abs(x - 0.5 * K.ones_like(x)))\n",
        "\n",
        "MAP_FUNC=kaneko_keras"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HD2eZiDd8kJL",
        "colab_type": "text"
      },
      "source": [
        "### Layer Utils\n",
        "- two-neighbor coupling\n",
        "- conv-to-dense block\n",
        "**N.B**: instead of updating the boundaries separately, just _wrap_ them. I.e., make L, C, and R all same-size, with L and R appropriately shifted. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2fgF7DUsdfcV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def two_neighbor_cml(inp):\n",
        "  x, a, strength = inp\n",
        "  num_features = K.int_shape(x)[-1]\n",
        "  batch_size = K.shape(x)[0]\n",
        "\n",
        "  # update the middle first\n",
        "  L = x[:, :-2]\n",
        "  C = x[:, 1:-1]\n",
        "  R = x[:, 2:]\n",
        "  a_mid = a[:, :-2]\n",
        "  str_mid = strength[:, :-2]\n",
        "  lattice_middle = (1.0 - str_mid) * MAP_FUNC(C, a_mid) + (str_mid/2.0) * (MAP_FUNC(L, a_mid) + MAP_FUNC(R, a_mid))\n",
        "\n",
        "  # then, update the boundaries\n",
        "  lattice_left = MAP_FUNC(x[:, 0], a[:, 0])\n",
        "  lattice_right = MAP_FUNC(x[:, -1], a[:, -1])\n",
        "\n",
        "  lattice_left = K.reshape(x=lattice_left, shape=(batch_size, 1))\n",
        "  lattice_right = K.reshape(x=lattice_right, shape=(batch_size, 1))\n",
        "\n",
        "  return K.concatenate([lattice_left, lattice_middle, lattice_right], axis=-1)\n",
        "\n",
        "\n",
        "def down_conv_block_with_pooling(inputs, num_output_features, base_sz=64, scope=\"default\"):\n",
        "  # 2x2 strided conv\n",
        "  conv_1 = layers.Conv2D(base_sz, (3, 3), strides=(1, 1), padding='same', name=scope + '-c1')(inputs)\n",
        "  bn_1 = conv_1#tf.layers.BatchNormalization(name=scope + '-bn1')(conv_1)\n",
        "  relu_1 = layers.ReLU(name=scope + '-lru1')(bn_1)\n",
        "  pool_1 = layers.MaxPooling2D((2, 2))(relu_1)\n",
        "  # 2x2 strided conv\n",
        "  conv_2 = layers.Conv2D(base_sz, (3, 3), strides=(1, 1), padding='same', name=scope + '-c2')(pool_1)\n",
        "  bn_2 = conv_2#layers.BatchNormalization(name=scope + '-bn2')(conv_2)\n",
        "  relu_2 = layers.ReLU(name=scope + '-lru2')(bn_2)\n",
        "  pool_2 = layers.MaxPooling2D((2, 2))(relu_2)\n",
        "  # flatten\n",
        "  flat = layers.Flatten(name=scope + '-flt1')(pool_2)\n",
        "  # dense\n",
        "  dense = layers.Dense(num_output_features, name=scope + '-d1')(flat)\n",
        "  drop = layers.Dropout(0.4)(dense)\n",
        "  return drop  \n",
        "\n",
        "\n",
        "def down_conv_block(inputs, num_output_features, base_sz=64, scope=\"default\"):\n",
        "\n",
        "  # 2x2 strided conv\n",
        "  conv_1 = layers.Conv2D(base_sz, (3, 3), strides=(2, 2), padding='same', name=scope + '-c1')(inputs)\n",
        "  bn_1 = conv_1#tf.layers.BatchNormalization(name=scope + '-bn1')(conv_1)\n",
        "  relu_1 = layers.LeakyReLU(name=scope + '-lru1')(bn_1)\n",
        "  # 2x2 strided conv\n",
        "  conv_2 = layers.Conv2D(base_sz, (3, 3), strides=(2, 2), padding='same', name=scope + '-c2')(relu_1)\n",
        "  bn_2 = conv_2#layers.BatchNormalization(name=scope + '-bn2')(conv_2)\n",
        "  relu_2 = layers.LeakyReLU(name=scope + '-lru2')(bn_2)\n",
        "  # flatten\n",
        "  flat = layers.Flatten(name=scope + '-flt1')(relu_2)\n",
        "  # dense\n",
        "  dense = layers.Dense(num_output_features, name=scope + '-d1')(flat)\n",
        "  drop = layers.Dropout(0.4)(dense)\n",
        "  return drop  \n",
        "\n",
        "def tile_param_along_lattice(inputs):\n",
        "  parameter, num_features = inputs\n",
        "  expanded = K.expand_dims(parameter)\n",
        "  batch_size = K.shape(parameter)[0]\n",
        "\n",
        "  tiled = K.tile(expanded, (batch_size, num_features))\n",
        "\n",
        "  return tiled"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z2XwXrM2VsOH",
        "colab_type": "text"
      },
      "source": [
        "### Two-Neighbor Coupled Map Lattice (CML)\n",
        "Just wrap the entire tf fucntion (slicing included) in a `Lambda` layer...\n",
        "- may need to apply the update in a loop... that should be fine\n",
        "- maybe a `TwoNeighborCoupled` layer? applied once, updates once?\n",
        "```\n",
        "dense = Flatten()(convs)\n",
        "for t in range(timesteps):\n",
        "    dense = TwoNeighborCML()(dense)\n",
        "```\n",
        "gradients might be a nightmare, tho kaneko's logistic map itself has a nicely simple gradient"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GWDvQtyDWeND",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def CML_NN(inputs, NUM_FEATURES=128, timesteps=1, n_classes=10):\n",
        "  \n",
        "  # shared conv features\n",
        "  shared_features = layers.Conv2D(64, (3, 3), padding='same', name=\"shared\")(inputs)\n",
        "  # get initial lattice conditions\n",
        "  initial_conditions = down_conv_block_with_pooling(shared_features, NUM_FEATURES, 64, \"init\")\n",
        "  # TESTING\n",
        "  # - sigmoid kills gradient\n",
        "  # - tanh+relu gets same uniform scale\n",
        "  initial_conditions = layers.Activation(INITIALIZATION_ACTIVATION, name='initial-conditions')(initial_conditions)\n",
        "\n",
        "  # get system params (B.P. + coupling strength)\n",
        "  system_params = down_conv_block(shared_features, 2, 64, \"sys\")#down_conv_block(shared_features, 2, 64, \"sys\")\n",
        "  bp = system_params[:, 0]\n",
        "  #coupling = layers.Activation(\"sigmoid\")(system_params[1][1])\n",
        "  coupling = layers.Activation(\"tanh\")(system_params[:, 1])\n",
        "  coupling = layers.Activation(\"relu\")(coupling)\n",
        "\n",
        "  # tile bp and coupling to the entire lattice!\n",
        "  #bp = layers.Lambda(tf.keras.backend.tile, arguments={'n':(-1, NUM_FEATURES)})(bp)\n",
        "  #coupling = layers.Lambda(tf.keras.backend.tile, arguments={'n':(-1, NUM_FEATURES)})(coupling)\n",
        "  #bp = layers.RepeatVector(NUM_FEATURES)(bp)\n",
        "  #coupling = layers.RepeatVector(NUM_FEATURES)(coupling)\n",
        "  bp = layers.Lambda(tile_param_along_lattice, name=\"bp-tiling\", output_shape=(NUM_FEATURES,))([bp, NUM_FEATURES])\n",
        "  coupling = layers.Lambda(tile_param_along_lattice, name=\"coupling-tiling\", output_shape=(NUM_FEATURES,))([coupling, NUM_FEATURES])\n",
        "\n",
        "  # for now, don't learn these.\n",
        "  bp = K.ones_like(initial_conditions)*BP#system_params[1][0]) * 1.69 #1.7522\n",
        "  coupling = K.ones_like(initial_conditions)*COUPLING#system_params[1][0]) * 0.5 #0.00115\n",
        "\n",
        "  system_state = initial_conditions\n",
        "  \n",
        "  # evolve CML\n",
        "  for i in range(timesteps):\n",
        "    system_state = layers.Lambda(two_neighbor_cml, output_shape=(NUM_FEATURES,), name=\"lattice-evolution-\" + str(i))([system_state, bp, coupling])\n",
        "  \n",
        "  # residual connection\n",
        "  system_state = layers.Add()([system_state, initial_conditions])#system_state + initial_conditions\n",
        "\n",
        "  #system_state = layers.Dense(128)(system_state)\n",
        "  #system_state = layers.Activation('relu')(system_state)\n",
        "  system_state = layers.Dropout(0.5)(system_state)\n",
        "  decision_dense = layers.Dense(n_classes)(system_state)\n",
        "  decision_classes = layers.Activation('softmax')(decision_dense)\n",
        "\n",
        "  return decision_classes\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ln7DfZGdXiKT",
        "colab_type": "text"
      },
      "source": [
        "## Training Utils"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AtiiB6BQ5IiA",
        "colab_type": "text"
      },
      "source": [
        "### Data Preprocessing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qtqX_S8DXjmW",
        "colab_type": "code",
        "outputId": "4e32713e-9bdd-4625-e8a5-b0bf442a9ab1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "# the data, split between train and test sets\n",
        "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
        "\n",
        "if K.image_data_format() == 'channels_first':\n",
        "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
        "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
        "    input_shape = (1, img_rows, img_cols)\n",
        "else:\n",
        "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
        "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
        "    input_shape = (img_rows, img_cols, 1)\n",
        "\n",
        "x_train = x_train.astype('float32')\n",
        "x_test = x_test.astype('float32')\n",
        "x_train /= 255\n",
        "x_test /= 255\n",
        "print('x_train shape:', x_train.shape)\n",
        "print(x_train.shape[0], 'train samples')\n",
        "print(x_test.shape[0], 'test samples')\n",
        "\n",
        "# convert class vectors to binary class matrices\n",
        "y_train = tf.keras.utils.to_categorical(y_train, num_classes)\n",
        "y_test = tf.keras.utils.to_categorical(y_test, num_classes)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11493376/11490434 [==============================] - 0s 0us/step\n",
            "x_train shape: (60000, 28, 28, 1)\n",
            "60000 train samples\n",
            "10000 test samples\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XZAM60MSbGU_",
        "colab_type": "text"
      },
      "source": [
        "### Callback\n",
        "Visualizes CML evolution for test data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "itXOtIw6bIBz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def plot_cml(cml_data, timesteps, epoch):\n",
        "\n",
        "  multiple = 5\n",
        "  num_subplots = cml_data.shape[-2]\n",
        "  fig, axs = plt.subplots(1, num_subplots)\n",
        "  fig.set_size_inches(18.5, 10.5)\n",
        "  \n",
        "  for i in range(num_subplots):\n",
        "    cml = cml_data[0, :, i, :]\n",
        "    # make the timesteps \"larger\" for easier viewing\n",
        "    # - (the resize method looks cooler but is more ambiguous)\n",
        "    #multiple = int(cml.shape[1] / cml.shape[0])\n",
        "    #cml = np.repeat(cml, repeats=multiple, axis=0)\n",
        "    cml = cv2.resize(cml, (1024,1024)) \n",
        "    axs[i].imshow((cml * 127.5 + 127.5), cmap='magma')\n",
        "    axs[i].axis('off')\n",
        "    \n",
        "  plt.savefig('tmp/{}.png'.format(epoch))\n",
        "\n",
        "  plt.close()\n",
        "\n",
        "\n",
        "class viz_cml(tf.keras.callbacks.Callback):\n",
        "    def __init__(self, model, X_test, timesteps):\n",
        "        self.model = model\n",
        "        self.x = X_test\n",
        "        self.time = timesteps\n",
        "\n",
        "    def on_train_begin(self, logs={}):\n",
        "        return\n",
        "\n",
        "    def on_train_end(self, logs={}):\n",
        "        return\n",
        "\n",
        "    def on_epoch_begin(self, epoch, logs={}):\n",
        "        return\n",
        "\n",
        "    def on_epoch_end(self, epoch, logs={}):\n",
        "        model = self.model\n",
        "        inp = model.input  # input placeholder\n",
        "        outputs = []\n",
        "        outputs.append(model.get_layer('initial-conditions').output)  # get output of N's layer\n",
        "        for i in range(self.time):\n",
        "          outputs.append(model.get_layer('lattice-evolution-' + str(i)).output)\n",
        "        functors = K.function([inp], [outputs])\n",
        "        layer_outs = functors([self.x, 1.])\n",
        "        layer_outs = np.array(layer_outs)\n",
        "        #print('\\r OUTPUTS : {}'.format(layer_outs.shape))\n",
        "        plot_cml(layer_outs, self.time, epoch)\n",
        "        \n",
        "        return\n",
        "\n",
        "    def on_batch_begin(self, batch, logs={}):\n",
        "        return\n",
        "\n",
        "    def on_batch_end(self, batch, logs={}):\n",
        "        return"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UlT7a_WiXk0W",
        "colab_type": "text"
      },
      "source": [
        "## Running Training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3iMPjbi9WRTf",
        "colab_type": "code",
        "outputId": "19713218-65bf-4369-ab19-36423af7a898",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# CMLnet model\n",
        "K.clear_session()\n",
        "cml_input = layers.Input(shape=(img_rows, img_cols, 1))\n",
        "cml_output = CML_NN(cml_input, 128, TIME)\n",
        "cml_model = tf.keras.Model(inputs=cml_input, outputs=cml_output)\n",
        "\n",
        "cml_model.summary()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_1 (InputLayer)            [(None, 28, 28, 1)]  0                                            \n",
            "__________________________________________________________________________________________________\n",
            "shared (Conv2D)                 (None, 28, 28, 64)   640         input_1[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "init-c1 (Conv2D)                (None, 28, 28, 64)   36928       shared[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "init-lru1 (ReLU)                (None, 28, 28, 64)   0           init-c1[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "max_pooling2d (MaxPooling2D)    (None, 14, 14, 64)   0           init-lru1[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "init-c2 (Conv2D)                (None, 14, 14, 64)   36928       max_pooling2d[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "init-lru2 (ReLU)                (None, 14, 14, 64)   0           init-c2[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "max_pooling2d_1 (MaxPooling2D)  (None, 7, 7, 64)     0           init-lru2[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "init-flt1 (Flatten)             (None, 3136)         0           max_pooling2d_1[0][0]            \n",
            "__________________________________________________________________________________________________\n",
            "init-d1 (Dense)                 (None, 128)          401536      init-flt1[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "dropout (Dropout)               (None, 128)          0           init-d1[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "initial-conditions (Activation) (None, 128)          0           dropout[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "tf_op_layer_Shape_2 (TensorFlow [(2,)]               0           initial-conditions[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "tf_op_layer_Shape_3 (TensorFlow [(2,)]               0           initial-conditions[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "tf_op_layer_Fill (TensorFlowOpL [(None, 128)]        0           tf_op_layer_Shape_2[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "tf_op_layer_Fill_1 (TensorFlowO [(None, 128)]        0           tf_op_layer_Shape_3[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "tf_op_layer_Mul (TensorFlowOpLa [(None, 128)]        0           tf_op_layer_Fill[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "tf_op_layer_Mul_1 (TensorFlowOp [(None, 128)]        0           tf_op_layer_Fill_1[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "lattice-evolution-0 (Lambda)    (None, 128)          0           initial-conditions[0][0]         \n",
            "                                                                 tf_op_layer_Mul[0][0]            \n",
            "                                                                 tf_op_layer_Mul_1[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "lattice-evolution-1 (Lambda)    (None, 128)          0           lattice-evolution-0[0][0]        \n",
            "                                                                 tf_op_layer_Mul[0][0]            \n",
            "                                                                 tf_op_layer_Mul_1[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "lattice-evolution-2 (Lambda)    (None, 128)          0           lattice-evolution-1[0][0]        \n",
            "                                                                 tf_op_layer_Mul[0][0]            \n",
            "                                                                 tf_op_layer_Mul_1[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "lattice-evolution-3 (Lambda)    (None, 128)          0           lattice-evolution-2[0][0]        \n",
            "                                                                 tf_op_layer_Mul[0][0]            \n",
            "                                                                 tf_op_layer_Mul_1[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "lattice-evolution-4 (Lambda)    (None, 128)          0           lattice-evolution-3[0][0]        \n",
            "                                                                 tf_op_layer_Mul[0][0]            \n",
            "                                                                 tf_op_layer_Mul_1[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "lattice-evolution-5 (Lambda)    (None, 128)          0           lattice-evolution-4[0][0]        \n",
            "                                                                 tf_op_layer_Mul[0][0]            \n",
            "                                                                 tf_op_layer_Mul_1[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "lattice-evolution-6 (Lambda)    (None, 128)          0           lattice-evolution-5[0][0]        \n",
            "                                                                 tf_op_layer_Mul[0][0]            \n",
            "                                                                 tf_op_layer_Mul_1[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "lattice-evolution-7 (Lambda)    (None, 128)          0           lattice-evolution-6[0][0]        \n",
            "                                                                 tf_op_layer_Mul[0][0]            \n",
            "                                                                 tf_op_layer_Mul_1[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "lattice-evolution-8 (Lambda)    (None, 128)          0           lattice-evolution-7[0][0]        \n",
            "                                                                 tf_op_layer_Mul[0][0]            \n",
            "                                                                 tf_op_layer_Mul_1[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "lattice-evolution-9 (Lambda)    (None, 128)          0           lattice-evolution-8[0][0]        \n",
            "                                                                 tf_op_layer_Mul[0][0]            \n",
            "                                                                 tf_op_layer_Mul_1[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "add (Add)                       (None, 128)          0           lattice-evolution-9[0][0]        \n",
            "                                                                 initial-conditions[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "dropout_2 (Dropout)             (None, 128)          0           add[0][0]                        \n",
            "__________________________________________________________________________________________________\n",
            "dense (Dense)                   (None, 10)           1290        dropout_2[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "activation_2 (Activation)       (None, 10)           0           dense[0][0]                      \n",
            "==================================================================================================\n",
            "Total params: 477,322\n",
            "Trainable params: 477,322\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yMTtTjyVm5eN",
        "colab_type": "text"
      },
      "source": [
        "### Option 1: `model.fit()`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vRYWxsPgVNgi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!mkdir tmp"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rFXYXbLYm8IR",
        "colab_type": "code",
        "outputId": "da67e199-5f22-4944-e83e-df8e7f300184",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# Compiling the init-only CML\n",
        "cml_model.compile(loss=tf.keras.losses.categorical_crossentropy,\n",
        "                  optimizer=tf.keras.optimizers.Adadelta(),\n",
        "                  metrics=['accuracy'])\n",
        "\n",
        "\"\"\"\n",
        "cml_model.compile(loss=tf.keras.losses.categorical_crossentropy,\n",
        "                  optimizer=LRMultiplier('adam',\n",
        "                                         {lyr: 0.5 for lyr in [l.name for l in cml_model.layers\n",
        "                                                               if \"sys\" in l.name]}),\n",
        "                  metrics=['accuracy'])\n",
        "\"\"\"\n",
        "train_history_cml = cml_model.fit(x_train, y_train,\n",
        "                batch_size=batch_size,\n",
        "                epochs=epochs,\n",
        "                verbose=1,\n",
        "                validation_data=(x_test, y_test))#,\n",
        "                #callbacks=viz_cml(cml_model, x_test[0:3], TIME))\n",
        "\n",
        "score = cml_model.evaluate(x_test, y_test, verbose=0)\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/100\n",
            "118/118 [==============================] - 6s 50ms/step - loss: 3.0843 - accuracy: 0.0973 - val_loss: 2.6036 - val_accuracy: 0.0882\n",
            "Epoch 2/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 2.9348 - accuracy: 0.1055 - val_loss: 2.4256 - val_accuracy: 0.1120\n",
            "Epoch 3/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 2.8096 - accuracy: 0.1199 - val_loss: 2.3155 - val_accuracy: 0.1395\n",
            "Epoch 4/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 2.7301 - accuracy: 0.1294 - val_loss: 2.2334 - val_accuracy: 0.1719\n",
            "Epoch 5/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 2.6463 - accuracy: 0.1409 - val_loss: 2.1548 - val_accuracy: 0.2079\n",
            "Epoch 6/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 2.5710 - accuracy: 0.1535 - val_loss: 2.0796 - val_accuracy: 0.2599\n",
            "Epoch 7/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 2.4817 - accuracy: 0.1742 - val_loss: 2.0170 - val_accuracy: 0.3129\n",
            "Epoch 8/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 2.4117 - accuracy: 0.1920 - val_loss: 1.9651 - val_accuracy: 0.3384\n",
            "Epoch 9/100\n",
            "118/118 [==============================] - 5s 43ms/step - loss: 2.3253 - accuracy: 0.2125 - val_loss: 1.9161 - val_accuracy: 0.3496\n",
            "Epoch 10/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 2.2403 - accuracy: 0.2395 - val_loss: 1.8686 - val_accuracy: 0.3636\n",
            "Epoch 11/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 2.1632 - accuracy: 0.2659 - val_loss: 1.8322 - val_accuracy: 0.3762\n",
            "Epoch 12/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 2.0824 - accuracy: 0.2914 - val_loss: 1.8026 - val_accuracy: 0.3857\n",
            "Epoch 13/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 2.0216 - accuracy: 0.3123 - val_loss: 1.7740 - val_accuracy: 0.3998\n",
            "Epoch 14/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.9662 - accuracy: 0.3338 - val_loss: 1.7489 - val_accuracy: 0.4115\n",
            "Epoch 15/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.9209 - accuracy: 0.3494 - val_loss: 1.7128 - val_accuracy: 0.4282\n",
            "Epoch 16/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.8797 - accuracy: 0.3657 - val_loss: 1.6800 - val_accuracy: 0.4432\n",
            "Epoch 17/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.8414 - accuracy: 0.3767 - val_loss: 1.6441 - val_accuracy: 0.4569\n",
            "Epoch 18/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.8106 - accuracy: 0.3871 - val_loss: 1.6143 - val_accuracy: 0.4706\n",
            "Epoch 19/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.7718 - accuracy: 0.4037 - val_loss: 1.5845 - val_accuracy: 0.4847\n",
            "Epoch 20/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.7426 - accuracy: 0.4144 - val_loss: 1.5541 - val_accuracy: 0.5005\n",
            "Epoch 21/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.7211 - accuracy: 0.4230 - val_loss: 1.5333 - val_accuracy: 0.5084\n",
            "Epoch 22/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.6874 - accuracy: 0.4332 - val_loss: 1.5044 - val_accuracy: 0.5201\n",
            "Epoch 23/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.6649 - accuracy: 0.4416 - val_loss: 1.4848 - val_accuracy: 0.5288\n",
            "Epoch 24/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 1.6408 - accuracy: 0.4548 - val_loss: 1.4524 - val_accuracy: 0.5448\n",
            "Epoch 25/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.6199 - accuracy: 0.4604 - val_loss: 1.4306 - val_accuracy: 0.5551\n",
            "Epoch 26/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.5979 - accuracy: 0.4678 - val_loss: 1.4119 - val_accuracy: 0.5634\n",
            "Epoch 27/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.5735 - accuracy: 0.4796 - val_loss: 1.3841 - val_accuracy: 0.5769\n",
            "Epoch 28/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.5556 - accuracy: 0.4850 - val_loss: 1.3660 - val_accuracy: 0.5876\n",
            "Epoch 29/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.5317 - accuracy: 0.4981 - val_loss: 1.3448 - val_accuracy: 0.5985\n",
            "Epoch 30/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 1.5146 - accuracy: 0.5019 - val_loss: 1.3319 - val_accuracy: 0.6022\n",
            "Epoch 31/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 1.4940 - accuracy: 0.5095 - val_loss: 1.3133 - val_accuracy: 0.6108\n",
            "Epoch 32/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.4794 - accuracy: 0.5154 - val_loss: 1.3041 - val_accuracy: 0.6136\n",
            "Epoch 33/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.4621 - accuracy: 0.5218 - val_loss: 1.2907 - val_accuracy: 0.6176\n",
            "Epoch 34/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.4416 - accuracy: 0.5314 - val_loss: 1.2700 - val_accuracy: 0.6266\n",
            "Epoch 35/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 1.4285 - accuracy: 0.5331 - val_loss: 1.2516 - val_accuracy: 0.6348\n",
            "Epoch 36/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.4059 - accuracy: 0.5451 - val_loss: 1.2386 - val_accuracy: 0.6387\n",
            "Epoch 37/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.3913 - accuracy: 0.5510 - val_loss: 1.2256 - val_accuracy: 0.6443\n",
            "Epoch 38/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.3811 - accuracy: 0.5564 - val_loss: 1.2107 - val_accuracy: 0.6495\n",
            "Epoch 39/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 1.3621 - accuracy: 0.5630 - val_loss: 1.1893 - val_accuracy: 0.6608\n",
            "Epoch 40/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 1.3509 - accuracy: 0.5659 - val_loss: 1.1788 - val_accuracy: 0.6645\n",
            "Epoch 41/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.3378 - accuracy: 0.5717 - val_loss: 1.1679 - val_accuracy: 0.6688\n",
            "Epoch 42/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 1.3235 - accuracy: 0.5801 - val_loss: 1.1590 - val_accuracy: 0.6723\n",
            "Epoch 43/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.3043 - accuracy: 0.5866 - val_loss: 1.1428 - val_accuracy: 0.6792\n",
            "Epoch 44/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.2959 - accuracy: 0.5907 - val_loss: 1.1259 - val_accuracy: 0.6877\n",
            "Epoch 45/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.2817 - accuracy: 0.5951 - val_loss: 1.1179 - val_accuracy: 0.6897\n",
            "Epoch 46/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 1.2702 - accuracy: 0.5990 - val_loss: 1.0993 - val_accuracy: 0.6994\n",
            "Epoch 47/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 1.2575 - accuracy: 0.6041 - val_loss: 1.0939 - val_accuracy: 0.6992\n",
            "Epoch 48/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.2397 - accuracy: 0.6118 - val_loss: 1.0833 - val_accuracy: 0.7025\n",
            "Epoch 49/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.2302 - accuracy: 0.6157 - val_loss: 1.0734 - val_accuracy: 0.7065\n",
            "Epoch 50/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.2190 - accuracy: 0.6219 - val_loss: 1.0601 - val_accuracy: 0.7118\n",
            "Epoch 51/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.2032 - accuracy: 0.6264 - val_loss: 1.0547 - val_accuracy: 0.7136\n",
            "Epoch 52/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 1.2054 - accuracy: 0.6261 - val_loss: 1.0338 - val_accuracy: 0.7231\n",
            "Epoch 53/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 1.1874 - accuracy: 0.6346 - val_loss: 1.0314 - val_accuracy: 0.7233\n",
            "Epoch 54/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 1.1758 - accuracy: 0.6395 - val_loss: 1.0204 - val_accuracy: 0.7261\n",
            "Epoch 55/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 1.1713 - accuracy: 0.6385 - val_loss: 1.0095 - val_accuracy: 0.7304\n",
            "Epoch 56/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.1591 - accuracy: 0.6449 - val_loss: 0.9942 - val_accuracy: 0.7363\n",
            "Epoch 57/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 1.1485 - accuracy: 0.6480 - val_loss: 0.9874 - val_accuracy: 0.7386\n",
            "Epoch 58/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 1.1397 - accuracy: 0.6507 - val_loss: 0.9829 - val_accuracy: 0.7396\n",
            "Epoch 59/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 1.1257 - accuracy: 0.6571 - val_loss: 0.9756 - val_accuracy: 0.7421\n",
            "Epoch 60/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 1.1202 - accuracy: 0.6597 - val_loss: 0.9609 - val_accuracy: 0.7499\n",
            "Epoch 61/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.1090 - accuracy: 0.6636 - val_loss: 0.9560 - val_accuracy: 0.7495\n",
            "Epoch 62/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.0979 - accuracy: 0.6674 - val_loss: 0.9457 - val_accuracy: 0.7548\n",
            "Epoch 63/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.0917 - accuracy: 0.6718 - val_loss: 0.9379 - val_accuracy: 0.7553\n",
            "Epoch 64/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.0843 - accuracy: 0.6747 - val_loss: 0.9294 - val_accuracy: 0.7576\n",
            "Epoch 65/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.0709 - accuracy: 0.6805 - val_loss: 0.9210 - val_accuracy: 0.7597\n",
            "Epoch 66/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 1.0652 - accuracy: 0.6833 - val_loss: 0.9148 - val_accuracy: 0.7607\n",
            "Epoch 67/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 1.0581 - accuracy: 0.6852 - val_loss: 0.9068 - val_accuracy: 0.7634\n",
            "Epoch 68/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.0460 - accuracy: 0.6887 - val_loss: 0.8977 - val_accuracy: 0.7669\n",
            "Epoch 69/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.0361 - accuracy: 0.6953 - val_loss: 0.8955 - val_accuracy: 0.7652\n",
            "Epoch 70/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.0269 - accuracy: 0.6982 - val_loss: 0.8840 - val_accuracy: 0.7719\n",
            "Epoch 71/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 1.0245 - accuracy: 0.6971 - val_loss: 0.8727 - val_accuracy: 0.7768\n",
            "Epoch 72/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.0131 - accuracy: 0.7037 - val_loss: 0.8706 - val_accuracy: 0.7753\n",
            "Epoch 73/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 1.0062 - accuracy: 0.7074 - val_loss: 0.8611 - val_accuracy: 0.7785\n",
            "Epoch 74/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.9974 - accuracy: 0.7092 - val_loss: 0.8532 - val_accuracy: 0.7809\n",
            "Epoch 75/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 0.9865 - accuracy: 0.7140 - val_loss: 0.8446 - val_accuracy: 0.7839\n",
            "Epoch 76/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 0.9776 - accuracy: 0.7173 - val_loss: 0.8373 - val_accuracy: 0.7868\n",
            "Epoch 77/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.9708 - accuracy: 0.7196 - val_loss: 0.8325 - val_accuracy: 0.7871\n",
            "Epoch 78/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.9629 - accuracy: 0.7218 - val_loss: 0.8198 - val_accuracy: 0.7930\n",
            "Epoch 79/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.9583 - accuracy: 0.7253 - val_loss: 0.8145 - val_accuracy: 0.7950\n",
            "Epoch 80/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.9520 - accuracy: 0.7269 - val_loss: 0.8101 - val_accuracy: 0.7973\n",
            "Epoch 81/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.9432 - accuracy: 0.7306 - val_loss: 0.8048 - val_accuracy: 0.8000\n",
            "Epoch 82/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.9395 - accuracy: 0.7315 - val_loss: 0.7965 - val_accuracy: 0.8033\n",
            "Epoch 83/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.9252 - accuracy: 0.7396 - val_loss: 0.7933 - val_accuracy: 0.8038\n",
            "Epoch 84/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 0.9214 - accuracy: 0.7388 - val_loss: 0.7867 - val_accuracy: 0.8043\n",
            "Epoch 85/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.9171 - accuracy: 0.7411 - val_loss: 0.7789 - val_accuracy: 0.8078\n",
            "Epoch 86/100\n",
            "118/118 [==============================] - 5s 43ms/step - loss: 0.9044 - accuracy: 0.7468 - val_loss: 0.7738 - val_accuracy: 0.8110\n",
            "Epoch 87/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.9009 - accuracy: 0.7474 - val_loss: 0.7648 - val_accuracy: 0.8126\n",
            "Epoch 88/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.8977 - accuracy: 0.7482 - val_loss: 0.7636 - val_accuracy: 0.8112\n",
            "Epoch 89/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.8928 - accuracy: 0.7518 - val_loss: 0.7546 - val_accuracy: 0.8160\n",
            "Epoch 90/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.8806 - accuracy: 0.7556 - val_loss: 0.7440 - val_accuracy: 0.8196\n",
            "Epoch 91/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.8797 - accuracy: 0.7558 - val_loss: 0.7464 - val_accuracy: 0.8190\n",
            "Epoch 92/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.8659 - accuracy: 0.7619 - val_loss: 0.7412 - val_accuracy: 0.8204\n",
            "Epoch 93/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.8657 - accuracy: 0.7616 - val_loss: 0.7292 - val_accuracy: 0.8273\n",
            "Epoch 94/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.8555 - accuracy: 0.7668 - val_loss: 0.7261 - val_accuracy: 0.8277\n",
            "Epoch 95/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 0.8506 - accuracy: 0.7676 - val_loss: 0.7202 - val_accuracy: 0.8313\n",
            "Epoch 96/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.8469 - accuracy: 0.7691 - val_loss: 0.7105 - val_accuracy: 0.8350\n",
            "Epoch 97/100\n",
            "118/118 [==============================] - 5s 41ms/step - loss: 0.8403 - accuracy: 0.7725 - val_loss: 0.7054 - val_accuracy: 0.8367\n",
            "Epoch 98/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.8349 - accuracy: 0.7721 - val_loss: 0.7023 - val_accuracy: 0.8366\n",
            "Epoch 99/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.8292 - accuracy: 0.7750 - val_loss: 0.6939 - val_accuracy: 0.8415\n",
            "Epoch 100/100\n",
            "118/118 [==============================] - 5s 42ms/step - loss: 0.8241 - accuracy: 0.7784 - val_loss: 0.6912 - val_accuracy: 0.8429\n",
            "Test loss: 0.6912243366241455\n",
            "Test accuracy: 0.8428999781608582\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SLp0pWFo9FL0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CVbF1o7699ON",
        "colab_type": "code",
        "outputId": "5c6192d0-91c7-494d-9574-eb5b7aa09596",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        }
      },
      "source": [
        "loss = train_history_cml.history['val_accuracy']\n",
        "#loss_alt = train_history_alt.history['val_loss']\n",
        "plt.plot(loss)\n",
        "#plt.plot(loss_alt)\n",
        "#plt.legend(['val loss (CML)', 'val loss (ReLU)'])\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3de3xV5Z3v8c8vVwghhEC45UIAQUBEgQDitGq99OCl4tTaIlOVjhU7p4xOx5mpvRzb0deZaWd6Om2nTCsq3lq1FrVlWpRaq9ZauQRQuUO4JoGQkDu57+R3/sjWiRgkwE52svb3/XrlZdZaj3v9liv5uvKs9TzL3B0REen/4qJdgIiIRIYCXUQkIBToIiIBoUAXEQkIBbqISEAkRGvHw4cP97y8vGjtXkSkX9q4ceMxd8/salvUAj0vL4+CgoJo7V5EpF8ys4Mn26YuFxGRgFCgi4gEhAJdRCQgFOgiIgGhQBcRCQgFuohIQCjQRUQCImrPoYuIxIrD1Y0cOFZPSXUjJdWNXD55BNOz0yO+HwW6iEgPaAm1s2ZbKU+uPcj6/ZUf2DYsNVmBLiLSV7g77xbX8Oe9Few/dpwDxxo4XNOIO8TFQW1jiJrGVnIyBvLV+ZO5IHsIWUMHMmrIAJIT4nukJgW6iEg3tba1s25fJS9vL+V3249ypKYJgOGpyYwbnsLsvAzizHB3EuKN+dNGcemkEcTHWa/Up0AXEenCrtI6/lR4jOZQG02t7Rw4Vs+ru8qoawqRnBDHJZMyueeT53L55BFkDEqKdrmAAl1E5AOq6lv4/su7+fm6g7R3euXy8NRkrp42iiunjOTjEzMZmNQz3SZnQ4EuIjGloSXEm4UVVDW0kJqcwKDkBBpb2iipbqS4qoEXNpdQ29jKrfPy+NKlE0hPSSQ5IQ6z3uk2ORsKdBEJvIaWEL959whrtpaGu1Hau2w3IDGO2XkZfOPaKUweldbLVZ49BbqI9GvuTnFVI3vLj3OosoGSqkZSkxPIGjqQYanJvLqzjOc2FVPXFCJ76EAWzc3lqikjyclI4XhziPrmEMkJ8WQNHcjQlMR+cSV+Mt0KdDObD/wQiAcedvfvnLA9F3gcSA+3udfdV0e4VhERAGqbWnlj9zHe2FPOG3uOUVLd+P62pPg4WtraP7B89fmj+Ku5Y5mdN7RfB/apnDLQzSweWAZcBRQDG8xslbtv79Tsm8Cz7v4TM5sKrAbyeqBeEQm4tnanqLKB+pYQjS1tNIS/GltDlNU289qucjYcqCTU7gxOTuDic4bxpUvHM3l0GrkZKYwYnExzqJ2S6kaO1jQxadRghqcmR/uwekV3rtDnAIXuvg/AzJ4BFgCdA92B9zqchgCHI1mkiASXu1NS3cjafZW8vrucN/aUU93QetL2k0am8sWPj+eKKSOYkZNOQvyHp6QakBjPhMxUJmSm9mTpfU53Aj0LKOq0XAzMPaHNt4HfmdnfAoOAK7v6IDNbAiwByM3NPd1aRSQAahpb2Xa4hm0ltbxdVM3Gg1WU1nYM0MkcnMwVk0cyd1wGaQMTSUmKJyUpnoFJ8aQkJZA2IIFhMXK1fSYidVP0ZuAxd/9/ZjYPeNLMprn7B24lu/tyYDlAfn6+d/E5IhIg7s5beyv4455j7D5ax67Sug/0d48ZMoDZ4zLIHzuU2XkZTBk9ONB93D2tO4FeAuR0Ws4Or+vsdmA+gLu/ZWYDgOFAWSSKFJH+xd15s7CCH/x+NwUHq0iMNyZkpjJz7FAWzc1lWtYQzhuTFjN9272lO4G+AZhoZuPoCPKFwKIT2hwCrgAeM7MpwACgPJKFikjf09TaxqZDVazbV8mGA5WU1zXT0tZOY0sbZXXNjEobwAMLzuOm/BwGJPa9kZVBc8pAd/eQmS0F1tDxSOIKd99mZvcDBe6+CrgHeMjMvkLHDdLF7q4uFZGAqW8OsaWkhrX7KnhrbwWbD1XT0tZOnMHUMWlMHJlKYnwcifFxXJCTzmfzs3tsZkH5MItW7ubn53tBQUFU9i0iH+bu7D9Wz9CUJNLDA2zK65p5s/AYbxYe4+2iagrLj+MOZjB1dBrzxg9j3oRh5OdlMGRgYrQPISaY2UZ3z+9qm0aKisS42qZWnt9YzJNrD7K3vB6A1OQEMgYlcaiyAYChKYnMzB3KtdNHc37WEGaNHUp6St+YYVD+hwJdJEY1tIR48PV9PPzGPupb2rggJ50HbphGc2sbxVWNlNc187nZOVwyMZPzxqQR10tzesuZU6CLBNyRmkZe2VHG3vLj5AxNYdzwQVTUt/C9NbsorW3imvNHceclE7ggJ/KvRJPepUAXCZhjx5spOFDFxoOVvFlYwfYjtQAkJ8R9YJbB87OG8J+LZjA7LyNapUqEKdBF+qHyumZ2HKllZ2ktO0vrKK5s5Fh9M5X1Le8Pm0+Kj+PCnHTuvXoyV04ZwYTMVCrrWzhQUU9jSzsXTximbpSAUaCL9FGtbe28vP0oBysaaGwJ0dDSxoGKeraW1L4/VB5gZFoyY4cNYsqoNIalJpE9dCCzxg5lWtaQDz0yOCw1WUPnA0yBLtIHVBxvpqGl7f3BNy9sLubRNw+8/xJigIGJ8YxJH8BF4zOYljWEqaPTmDw6rc+8z1KiT4EuEkXNoTZ+9Moefvr6PtraPzgmZN74YfzLX57P3PEZDEiIV/eInJICXSRK3imq5h9XvsPuo8e5cWY2c8dn0NzaRnOonYvGD2Na1pBolyj9jAJdpAc1tbZRWHacY8ebqTjeQmltE1uKa3i7qJrS2iZGpQ3g0cWz+cTkEdEuVQJAgS4SYRsOVPKLDUVsLalhT9nxD3WljB2WwpxxGczITefTM7M1ZF4iRoEuEiEFByr5we/38KfCYwwZmMjM3HSumjqSKaPTGJk2gOGpSQxLTSY1Wb920jP0kyVyFiqON7PqncM8v6mELSU1DE9N4pvXTuGv5o5lYJJmGZTepUAXOYVQWzvFVY2E2p22dudwTSMb9nfM/735UDWhdue8MWl861NT+dzsHFKS9Gsl0aGfPJGPcLCiniVPbGTX0boPrE+IM87LGsIdl4znhguzOHfU4ChVKPI/FOgiJ/HGnnKWPrUZgPsXnMeQgYkkxMUxdFAiF+ak60pc+hz9RIqEVTe0sLWklr3lx9lxpJZnC4qYOGIwD92aT+6wlGiXJ3JKCnSJeXvLj/PIn/bz3Mbi92cjTE1O4IYLs3jghmkM0lMp0k906yfVzOYDP6TjnaIPu/t3Ttj+H8AnwospwAh31+TK0mfVNbXy+x1H+fXbh3ltVzlJCXHcODOLT00fw4QRqYwYnIyZhtpL/3LKQDezeGAZcBVQDGwws1Xuvv29Nu7+lU7t/xaY0QO1ipyVosoG/rinnNd2lfP67nJaQu2MGTKAu66YyK3zxjJcsxBKP9edK/Q5QKG77wMws2eABcD2k7S/GfhWZMoTOTMFByr5p+fepbaxlfg4wx3K6poByEofyKI5uXzqgtHMyBmqSa8kMLoT6FlAUaflYmBuVw3NbCwwDvjDSbYvAZYA5ObmnlahIt317IYivvGrLYxJH8gnzxtFW5sTanemjB7MZeeOYELmIHWnSCBF+m7PQmClu7d1tdHdlwPLAfLz872rNiJnor45xJ6y4/xqcwmP/fkAHztnOD9eNENvppeY0p1ALwFyOi1nh9d1ZSHw5bMtSqS7Xtp6hO++tIv9x+rfX7f44jy+ee0UEuLjoliZSO/rTqBvACaa2Tg6gnwhsOjERmY2GRgKvBXRCkW60NjSxgO/3c5T6w4xdXQa91w1iUmjBjNlVJqeGZeYdcpAd/eQmS0F1tDx2OIKd99mZvcDBe6+Ktx0IfCMu6srRXrUpkNVfHXlu+wpO86dl47nnqvOJSlBV+Mi3epDd/fVwOoT1t13wvK3I1eWyIeV1TXx3Rd38dymYkamJfPk7XP4+MTMaJcl0mdoCJz0aQ0tIf605xh/2FnGb949Qkuonb+5bAJLP3GORnCKnEC/EdLntITaeW1XGc9vKuEPu8poCbUzODmBK6aM4O+unMS44YOiXaJIn6RAlz6hOdTGn/dW8LttR3lp6xGqGloZnprEojm5fHLqSGaPyyBRT62IfCQFukSVu/Nfr+3lJ6/t5XhziEFJ8Vw+ZSSfnpHFxycO16OHIqdBgS5R4+78y+odPPTGfq6aOpJFc3K5+JxhJCfo1W0iZ0KBLlHR3u58a9U2nlx7kNvmjeVbnzpPc6qInCUFuvQqd2f9/kr+67W9vL67nDsvGc+9V0/W3CoiEaBAl14Ramvnt1uO8PAb+9lSUkN6SiLfvHYKt39snMJcJEIU6NKjmkNtPL+phJ++vpeDFQ2MzxzE//3LaXx6RjYDk9RXLhJJCnSJuFBbO2/tq2D1liOs2XaUyvoWpmcP4cFbZnHVlJHqKxfpIQp0iai39lbwD798h5LqRgYlxXPFlJHclJ/Nx84Zrq4VkR6mQJeIaA618b01u3j4T/vJGzaIn35+JpedO4IBiepWEektCnQ5a/vKj/O/f76JnaV1fP6iXL5+zRRSkvSjJdLb9FsnZ2X1liP808p3SYw3VizO5/LJI6NdkkjMUqDLGTlwrJ4Vb+7nibcOMiM3nWWLZjImfWC0yxKJaQp06ZbWtna2Ha7lrb0V/HbLYbaW1AIdr3v7+jVT9IIJkT5AgS4faceRWv71xZ0UHKikoaXj3d8X5KTzzWuncM35o3VVLtKHdCvQzWw+8EM6XkH3sLt/p4s2nwW+DTjwjrt/6L2j0n+4Oz9be5AHfruDtAGJ3DQrm9njMpiTl8GItAHRLk9EunDKQDezeGAZcBVQDGwws1Xuvr1Tm4nA14C/cPcqMxvRUwVLz3J3tpTU8OM/FPK77Ue57NxMvnfTBQxPTY52aSJyCt25Qp8DFLr7PgAzewZYAGzv1OYOYJm7VwG4e1mkC5WeVVbXxH+9upffbSvlcE0TifHGN67pmGtFIztF+ofuBHoWUNRpuRiYe0KbSQBm9iYd3TLfdveXTvwgM1sCLAHIzc09k3qlBxyqaODzj6yjtLaJSydl8pWrJnHllJEMHZQU7dJE5DRE6qZoAjARuAzIBv5oZue7e3XnRu6+HFgOkJ+f7xHat5yFXaV13PLIOlra2vnFkouYkTs02iWJyBnqzrNmJUBOp+Xs8LrOioFV7t7q7vuB3XQEvPRhmw5V8dkH38IMnr1znsJcpJ/rTqBvACaa2TgzSwIWAqtOaPMrOq7OMbPhdHTB7ItgnRJhr+4qY9FDa0lPSWTlly5m0sjB0S5JRM7SKQPd3UPAUmANsAN41t23mdn9ZnZ9uNkaoMLMtgOvAv/o7hU9VbScnRc2F3PH4wVMyExl5ZcuJicjJdoliUgEmHt0urLz8/O9oKAgKvuOZU+tO8TXX9jCvPHDWH7rLAYPSIx2SSJyGsxso7vnd7VNI0VjyCs7jvLNX23hsnMz+ennZ2lqW5GA0QQcMWJLcQ1Ln9rM1DFpLFs0U2EuEkAK9BhQXNXAXz++gYxBSay4bTaDkvWHmUgQ6Tc74F7efpSvPvcurW3t/PyLczUPi0iAKdADqqElxAO/2cHT6w9x3pg0frjwQs4ZoUcTRYJMgR5ApTVNLH50PbuO1nHnpeO556pzNV+5SAxQoAfMztJavvDoBuqaQjz2hTlcOikz2iWJSC9RoAfIn/ce484nNpKSHM+zd85j6pi0aJckIr1IgR4QRZUN3PnkRkYNGcDjfz1HbxISiUHqWA2A1rZ27npmMzisWDxbYS4So3SFHgDff3k3mw9V8+NFMzQvi0gM0xV6P/fGnnJ+8tpebp6Tw3XTx0S7HBGJIgV6P/bcxmKWPLGRiSNSue+686JdjohEmbpc+qGGlhD3/XobKzcWM2dcBj9aOIOBSZqbRSTWKdD7mfK6Zm55ZB27jtZx1xUTuevyc0iI1x9aIqJA71fK65pZ9NBaiqoaNGhIRD5Egd5PlNU1seihdZRUNfLo4jnMmzAs2iWJSB/Trb/VzWy+me0ys0Izu7eL7YvNrNzM3g5/fTHypcaud4qq+dyDazvC/AuzFeYi0qVTXqGbWTywDLgKKAY2mNkqd99+QtNfuPvSHqgxZjWH2vjPVwr5yet7yUxN5onb5zA7LyPaZYlIH9WdLpc5QKG77wMws2eABcCJgS4RVNfUysLla9l2uJbPzMrm/1w3lSED9f5PETm57gR6FlDUabkYmNtFuxvN7BJgN/AVdy86sYGZLQGWAOTm5p5+tTHC3fna81vYWVrHTz8/i/nTRkW7JBHpByL1vNt/A3nuPh14GXi8q0buvtzd8909PzNTT2iczNPri/jNu0e455OTFOYi0m3dCfQSIKfTcnZ43fvcvcLdm8OLDwOzIlNe7NlxpJZ//u9tfHzicL50yYRolyMi/Uh3An0DMNHMxplZErAQWNW5gZmN7rR4PbAjciXGjuPNIZY+tYm0gYn8x+cuJC7Ool2SiPQjp+xDd/eQmS0F1gDxwAp332Zm9wMF7r4KuMvMrgdCQCWwuAdrDqS2dufupzdzoKKBn90+l+GpydEuSUT6mW4NLHL31cDqE9bd1+n7rwFfi2xpseW7L+3klZ1lPLDgPD1nLiJnRJOA9AG/LChi+R/3ceu8sdwyLy/a5YhIP6VAj7J1+yr4+gtb+Ng5w7nvuqnRLkdE+jEFehTtP1bPnT/bSE5GCssWzdSsiSJyVpQgUVJV38IXHl1PnBmPLZ7DkBSNAhWRs6PZFqOgOdTGnU9u5HBNE0/fMZfcYXoPqIicPV2h9zJ3597ntrD+QCXfu+kCZo3VZFsiEhkK9F72w1f28MLmEv7hk5O4/gK91FlEIkeB3ote2FzMD36/hxtnZvPlT5wT7XJEJGAU6L1k48FKvrpyCxeNz+BfP30+ZhrWLyKRpUDvBWV1TfzNzzYxOn0AD34+n6QE/WcXkchTsvSw1rZ2lv58M7VNrTx4yyw9nigiPUaPLfawf129k/UHKvnB5y5k8qi0aJcjIgGmK/QetHrLEVa8uZ/FF+dxw4ysaJcjIgGnQO8hR2oa+drzW7ggewhfv2ZKtMsRkRigQO8B7e3OP/zyHVpC7fxg4QzdBBWRXqGk6QGP/vkAbxZWcN+npjJu+KBolyMiMUKBHmHbD9fy3Zd2cuWUkSycnXPqf0FEJEK6FehmNt/MdplZoZnd+xHtbjQzN7P8yJXYf+w5WsetK9aRPjCR79yowUMi0rtOGehmFg8sA64GpgI3m9mH3sRgZoOBu4F1kS6yP9h9tI6bH1qLmfHUHRfpnaAi0uu6c4U+Byh0933u3gI8Ayzoot0DwHeBpgjW1y/sOVrHoofWEmfGM0su4pwRqdEuSURiUHcCPQso6rRcHF73PjObCeS4+28/6oPMbImZFZhZQXl5+WkX2xcdO97M4kc3YGY8veQiJmQqzEUkOs76pqiZxQHfB+45VVt3X+7u+e6en5mZeba7jrrmUBtfenIjFfXNPHJbvsJcRKKqO4FeAnR+XCM7vO49g4FpwGtmdgC4CFgV9Buj7s7Xn99KwcEqvnfTBUzPTo92SSIS47oT6BuAiWY2zsySgIXAqvc2unuNuw939zx3zwPWAte7e0GPVNwHuDvff3k3z20q5u+unMh10/WiChGJvlNOzuXuITNbCqwB4oEV7r7NzO4HCtx91Ud/QrDUN4f4x5XvsHpLKTfNyubuKyZGuyQREaCbsy26+2pg9Qnr7jtJ28vOvqy+qaiygTueKGD30Tq+fs1k7vj4eD1rLiJ9hqbP7abiqgY+++Bb1DeHePQLc7h0Uv+/qSsiwaJA74byumZueWQ99c0hfnHnPKaM1rzmItL3KNBPoaaxldtWrKe0pomffXGOwlxE+ixNzvURmkNt3PFEAXvK6vjpLbOYNTYj2iWJiJyUrtBPwt35xgtbWb+/kh8uvFB95iLS5+kK/SSW/3EfKzcWc/cVE1lwoV4fJyJ9nwK9Cy9vP8p3XtrJtdNH6zlzEek3FOgneHVnGUuf2sT5WUP43mcuIC5Oz5mLSP+gQO9k9ZYjLHmygEkjB/P4F+YwMCk+2iWJiHSbboqGvbC5mHuefYcZuUN59AuzSRuQGO2SREROiwIdKKtt4qsrtzB33DAeWZxPSpL+s4hI/6MuF+DRPx8g1N7Od248X2EuIv1WzAd6XVMrP1t7kKunjWbssEHRLkdE5IzFfKA/s76IuqYQSy4ZH+1SRETOSkwHekuonRVv7mfe+GFckKM3DolI/xbTgf7f7xzmSE0Td16qq3MR6f9iNtDdneV/3MfkUYM1T4uIBEK3At3M5pvZLjMrNLN7u9j+JTPbYmZvm9mfzGxq5EuNrOKqRnYdrePmObl665CIBMIpA93M4oFlwNXAVODmLgL7KXc/390vBP4N+H7EK42wTYeqAMjPGxrlSkREIqM7V+hzgEJ33+fuLcAzwILODdy9ttPiIMAjV2LP2HyompSkeM4dOTjapYiIRER3RtFkAUWdlouBuSc2MrMvA38PJAGXd/VBZrYEWAKQm5t7urVG1KZDVUzPHkJCfMzeRhCRgIlYmrn7MnefAHwV+OZJ2ix393x3z8/MjN6NyKbWNrYfrmVmrrpbRCQ4uhPoJUBOp+Xs8LqTeQa44WyK6mlbSmoItTszFOgiEiDdCfQNwEQzG2dmScBCYFXnBmbW+S0Q1wJ7Ildi5G062HFDdEauBhOJSHCcsg/d3UNmthRYA8QDK9x9m5ndDxS4+ypgqZldCbQCVcBtPVn02dp0qIqxw1IYnpoc7VJERCKmW1MLuvtqYPUJ6+7r9P3dEa6rx7g7mw5V8xcThkW7FBGRiIq5RzxKqhspr2tm5lj1n4tIsMRcoG86VA2gJ1xEJHBiL9APVjEgMY5zR2lAkYgES8wF+uaiaqZnp5OoAUUiEjAxlWodA4pq1N0iIoEUU4G+taSG1jbX8+ciEkgxFegbwwOKZukJFxEJoJgL9DwNKBKRgIqZQO8YUFSl/nMRCayYCfSiykaOHW/RgCIRCayYCfSNhyoB9Z+LSHDFTqAfrCI1OYFJekORiARUDAV6NRfmpBMfpxdCi0gwxUSgH28Osau0Vv3nIhJoMRHo7xRV0+7qPxeRYIuJQN94sAozuDBHI0RFJLhiJtAnjRjMkIGJ0S5FRKTHdCvQzWy+me0ys0Izu7eL7X9vZtvN7F0ze8XMxka+1DPT3h4eUDRWV+ciEmynDHQziweWAVcDU4GbzWzqCc02A/nuPh1YCfxbpAs9UztKa6lrCmmEqIgEXneu0OcAhe6+z91bgGeABZ0buPur7t4QXlwLZEe2zDO3ZmspcQaXnTsi2qWIiPSo7gR6FlDUabk4vO5kbgdePJuiIunFraXMzssgc7Am5BKRYIvoTVEz+zyQD/z7SbYvMbMCMysoLy+P5K67VFhWx56y41xz/uge35eISLR1J9BLgJxOy9nhdR9gZlcC3wCud/fmrj7I3Ze7e76752dmZp5JvaflxS2lAPyv80b1+L5ERKKtO4G+AZhoZuPMLAlYCKzq3MDMZgAP0hHmZZEv88y8uLWUWWOHMmrIgGiXIiLS404Z6O4eApYCa4AdwLPuvs3M7jez68PN/h1IBX5pZm+b2aqTfFyvOVhRz/YjtVw9TVfnIhIbErrTyN1XA6tPWHdfp++vjHBdZ+3FrR3dLfMV6CISIwI7UvTFLUeYnj2E7KEp0S5FRKRXBDLQS6obeae4hqun6ekWEYkdgQz0X7/d8RDONeeru0VEYkfgAt3deWFTCbPGDmXssEHRLkdEpNcELtC3Ha5lT9lx/nLGRw1mFREJnsAF+gubS0iMN66brv5zEYktgQr0UFs7v377MJ84dwTpKUnRLkdEpFcFKtD/VHiMY8eb+fRMdbeISOwJVKC/sLmEIQMT+cRkTZUrIrEnMIF+vDnEmm2lXDt9NMkJ8dEuR0Sk1wUm0H/77mGaWtv5tJ5uEZEYFYhAd3ce+/NBJo8azKyxetWciMSmQAT6+v2V7DhSy20X52Fm0S5HRCQqAhHoj791gCEDE7nhQnW3iEjs6veBfri6kTXbjvK52TkMTNLNUBGJXf0+0H++7iDt7txy0dholyIiElX9OtCbWtt4en0RV04ZSU6G5j0XkdjWrwN91duHqaxvYfHFedEuRUQk6roV6GY238x2mVmhmd3bxfZLzGyTmYXM7DORL/PDQm3tLHutkGlZaVw8YVhv7FJEpE87ZaCbWTywDLgamArcbGZTT2h2CFgMPBXpAk/m128f5mBFA3ddPlGPKoqI0L2XRM8BCt19H4CZPQMsALa/18DdD4S3tfdAjR/S1u78+NVCpoxO46qpI3tjlyIifV53ulyygKJOy8XhdafNzJaYWYGZFZSXl5/JRwDwm3cPs/9YPXdfcY6uzkVEwnr1pqi7L3f3fHfPz8zMPKPPaGt3fvTKHs4dOZhPTtU7Q0VE3tOdQC8BcjotZ4fXRcXqLUfYW17PXVdMJC5OV+ciIu/pTqBvACaa2TgzSwIWAqt6tqyTG5Qcz1VTR3L1NF2di4h0dsqbou4eMrOlwBogHljh7tvM7H6gwN1Xmdls4AVgKPApM/tndz+vJwq+fPJILp+sG6EiIifqzlMuuPtqYPUJ6+7r9P0GOrpiREQkSvr1SFEREfkfCnQRkYBQoIuIBIQCXUQkIBToIiIBoUAXEQkIBbqISECYu0dnx2blwMEz/NeHA8ciWE5/EYvHHYvHDLF53LF4zHD6xz3W3bucDCtqgX42zKzA3fOjXUdvi8XjjsVjhtg87lg8ZojscavLRUQkIBToIiIB0V8DfXm0C4iSWDzuWDxmiM3jjsVjhgged7/sQxcRkQ/rr1foIiJyAgW6iEhA9LtAN7P5ZrbLzArN7N5o19MTzCzHzF41s+1mts3M7g6vzzCzl81sT/ifQ6Nda6SZWbyZbTaz34SXx5nZuvD5/kX4rVmBYmbpZrbSzHaa2Q4zmxcj5/or4Z/vrWb2tJkNCNr5NrMVZlZmZls7revy3FqHH4WP/V0zm3m6++tXgW5m8cAy4GpgKnCzmU2NblU9IgTc4+5TgYuAL4eP817gFXefCLwSXg6au4EdnZa/C/yHu58DVAG3R/dDXPIAAALBSURBVKWqnvVD4CV3nwxcQMfxB/pcm1kWcBeQ7+7T6Hgb2kKCd74fA+afsO5k5/ZqYGL4awnwk9PdWb8KdGAOUOju+9y9BXgGWBDlmiLO3Y+4+6bw93V0/IJn0XGsj4ebPQ7cEJ0Ke4aZZQPXAg+Hlw24HFgZbhLEYx4CXAI8AuDuLe5eTcDPdVgCMNDMEoAU4AgBO9/u/keg8oTVJzu3C4AnvMNaIN3MRp/O/vpboGcBRZ2Wi8PrAsvM8oAZwDpgpLsfCW8qBYL2ctUfAP8EtIeXhwHV7h4KLwfxfI8DyoFHw11ND5vZIAJ+rt29BPgecIiOIK8BNhL88w0nP7dnnW/9LdBjipmlAs8Bf+futZ23ecfzpoF55tTMrgPK3H1jtGvpZQnATOAn7j4DqOeE7pWgnWuAcL/xAjr+hzYGGMSHuyYCL9Lntr8FegmQ02k5O7wucMwskY4w/7m7Px9effS9P8HC/yyLVn094C+A683sAB1daZfT0becHv6THIJ5vouBYndfF15eSUfAB/lcA1wJ7Hf3cndvBZ6n42cg6OcbTn5uzzrf+lugbwAmhu+EJ9FxE2VVlGuKuHDf8SPADnf/fqdNq4Dbwt/fBvy6t2vrKe7+NXfPdvc8Os7rH9z9r4BXgc+EmwXqmAHcvRQoMrNzw6uuALYT4HMddgi4yMxSwj/v7x13oM932MnO7Srg1vDTLhcBNZ26ZrrH3fvVF3ANsBvYC3wj2vX00DF+jI4/w94F3g5/XUNHn/IrwB7g90BGtGvtoeO/DPhN+PvxwHqgEPglkBzt+nrgeC8ECsLn+1fA0Fg418A/AzuBrcCTQHLQzjfwNB33CFrp+Gvs9pOdW8DoeIpvL7CFjieATmt/GvovIhIQ/a3LRURETkKBLiISEAp0EZGAUKCLiASEAl1EJCAU6CIiAaFAFxEJiP8PY1wcHgiijUYAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xzPNgOUutYAv",
        "colab_type": "text"
      },
      "source": [
        "### Generate CML GIF"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YoR-aPHmtfru",
        "colab_type": "code",
        "outputId": "1126c2d3-96c9-46f9-c458-cbfd57273683",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 358
        }
      },
      "source": [
        "import imageio\n",
        "\n",
        "dirName = \"tmp/\"\n",
        "\n",
        "gif_images = [imageio.imread(dirName + str(i) + '.png')  for i in range(epochs)]\n",
        "imageio.mimsave('cml.gif', gif_images)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "error",
          "ename": "FileNotFoundError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-14-5ce5c6e9433d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mdirName\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"tmp/\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mgif_images\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mimageio\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mimread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdirName\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'.png'\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mepochs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0mimageio\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmimsave\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'cml.gif'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgif_images\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-14-5ce5c6e9433d>\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mdirName\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"tmp/\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mgif_images\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mimageio\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mimread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdirName\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'.png'\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mepochs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0mimageio\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmimsave\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'cml.gif'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgif_images\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/imageio/core/functions.py\u001b[0m in \u001b[0;36mimread\u001b[0;34m(uri, format, **kwargs)\u001b[0m\n\u001b[1;32m    219\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    220\u001b[0m     \u001b[0;31m# Get reader and read first\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 221\u001b[0;31m     \u001b[0mreader\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0muri\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mformat\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"i\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    222\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mreader\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    223\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mreader\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/imageio/core/functions.py\u001b[0m in \u001b[0;36mget_reader\u001b[0;34m(uri, format, mode, **kwargs)\u001b[0m\n\u001b[1;32m    128\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    129\u001b[0m     \u001b[0;31m# Create request object\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 130\u001b[0;31m     \u001b[0mrequest\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mRequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0muri\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"r\"\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    131\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    132\u001b[0m     \u001b[0;31m# Get format\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/imageio/core/request.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, uri, mode, **kwargs)\u001b[0m\n\u001b[1;32m    123\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    124\u001b[0m         \u001b[0;31m# Parse what was given\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 125\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parse_uri\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0muri\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    126\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    127\u001b[0m         \u001b[0;31m# Set extension\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/imageio/core/request.py\u001b[0m in \u001b[0;36m_parse_uri\u001b[0;34m(self, uri)\u001b[0m\n\u001b[1;32m    271\u001b[0m                 \u001b[0;31m# Reading: check that the file exists (but is allowed a dir)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    272\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexists\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 273\u001b[0;31m                     \u001b[0;32mraise\u001b[0m \u001b[0mFileNotFoundError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"No such file: '%s'\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    274\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    275\u001b[0m                 \u001b[0;31m# Writing: check that the directory to write to does exist\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mFileNotFoundError\u001b[0m: No such file: '/content/tmp/0.png'"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4_UoDne3hWjU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}