multi_class_classification_of_handwritten_digits.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "multi-class_classification_of_handwritten_digits.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [
        "JndnmDMp66FL",
        "266KQvZoMxMv",
        "6sfw3LH0Oycm"
      ],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "[View in Colaboratory](https://colab.research.google.com/github/ArunkumarRamanan/Exercises-Machine-Learning-Crash-Course-Google-Developers/blob/master/multi_class_classification_of_handwritten_digits.ipynb)"
      ]
    },
    {
      "metadata": {
        "id": "JndnmDMp66FL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Copyright 2017 Google LLC."
      ]
    },
    {
      "metadata": {
        "id": "hMqWDc_m6rUC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "mPa95uXvcpcn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Classifying Handwritten Digits with Neural Networks"
      ]
    },
    {
      "metadata": {
        "id": "Fdpn8b90u8Tp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "![img](https://www.tensorflow.org/versions/r0.11/images/MNIST.png)"
      ]
    },
    {
      "metadata": {
        "id": "c7HLCm66Cs2p",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Learning Objectives:**\n",
        "  * Train both a linear model and a neural network to classify handwritten digits from the classic [MNIST](http://yann.lecun.com/exdb/mnist/) data set\n",
        "  * Compare the performance of the linear and neural network classification models\n",
        "  * Visualize the weights of a neural-network hidden layer"
      ]
    },
    {
      "metadata": {
        "id": "HSEh-gNdu8T0",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Our goal is to map each input image to the correct numeric digit. We will create a NN with a few hidden layers and a Softmax layer at the top to select the winning class."
      ]
    },
    {
      "metadata": {
        "id": "2NMdE1b-7UIH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Setup\n",
        "\n",
        "First, let's download the data set, import TensorFlow and other utilities, and load the data into a *pandas* `DataFrame`. Note that this data is a sample of the original MNIST training data; we've taken 20000 rows at random."
      ]
    },
    {
      "metadata": {
        "id": "4LJ4SD8BWHeh",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from __future__ import print_function\n",
        "\n",
        "import glob\n",
        "import math\n",
        "import os\n",
        "\n",
        "from IPython import display\n",
        "from matplotlib import cm\n",
        "from matplotlib import gridspec\n",
        "from matplotlib import pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import seaborn as sns\n",
        "from sklearn import metrics\n",
        "import tensorflow as tf\n",
        "from tensorflow.python.data import Dataset\n",
        "\n",
        "tf.logging.set_verbosity(tf.logging.ERROR)\n",
        "pd.options.display.max_rows = 10\n",
        "pd.options.display.float_format = '{:.1f}'.format\n",
        "\n",
        "mnist_dataframe = pd.read_csv(\n",
        "  \"https://dl.google.com/mlcc/mledu-datasets/mnist_train_small.csv\",\n",
        "  sep=\",\",\n",
        "  header=None)\n",
        "\n",
        "# Use just the first 10,000 records for training/validation.\n",
        "mnist_dataframe = mnist_dataframe.head(10000)\n",
        "\n",
        "mnist_dataframe = mnist_dataframe.reindex(np.random.permutation(mnist_dataframe.index))\n",
        "mnist_dataframe.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "kg0-25p2mOi0",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Each row represents one labeled example. Column 0 represents the label that a human rater has assigned for one handwritten digit. For example, if Column 0 contains '6', then a human rater interpreted the handwritten character as the digit '6'.  The ten digits 0-9 are each represented, with a unique class label for each possible digit. Thus, this is a multi-class classification problem with 10 classes."
      ]
    },
    {
      "metadata": {
        "id": "PQ7vuOwRCsZ1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "![img](https://www.tensorflow.org/versions/r0.11/images/MNIST-Matrix.png)"
      ]
    },
    {
      "metadata": {
        "id": "dghlqJPIu8UM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Columns 1 through 784 contain the feature values, one per pixel for the 28×28=784 pixel values. The pixel values are on a gray scale in which 0 represents white, 255 represents black, and values between 0 and 255 represent shades of gray. Most of the pixel values are 0; you may want to take a minute to confirm that they aren't all 0.  For example, adjust the following text block to print out the values in column 72."
      ]
    },
    {
      "metadata": {
        "id": "2ZkrL5MCqiJI",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "mnist_dataframe.loc[:, 72:72]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "vLNg2VxqhUZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now, let's parse out the labels and features and look at a few examples. Note the use of `loc` which allows us to pull out columns based on original location, since we don't have a header row in this data set."
      ]
    },
    {
      "metadata": {
        "id": "JfFWWvMWDFrR",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def parse_labels_and_features(dataset):\n",
        "  \"\"\"Extracts labels and features.\n",
        "  \n",
        "  This is a good place to scale or transform the features if needed.\n",
        "  \n",
        "  Args:\n",
        "    dataset: A Pandas `Dataframe`, containing the label on the first column and\n",
        "      monochrome pixel values on the remaining columns, in row major order.\n",
        "  Returns:\n",
        "    A `tuple` `(labels, features)`:\n",
        "      labels: A Pandas `Series`.\n",
        "      features: A Pandas `DataFrame`.\n",
        "  \"\"\"\n",
        "  labels = dataset[0]\n",
        "\n",
        "  # DataFrame.loc index ranges are inclusive at both ends.\n",
        "  features = dataset.loc[:,1:784]\n",
        "  # Scale the data to [0, 1] by dividing out the max value, 255.\n",
        "  features = features / 255\n",
        "\n",
        "  return labels, features"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "mFY_-7vZu8UU",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "training_targets, training_examples = parse_labels_and_features(mnist_dataframe[:7500])\n",
        "training_examples.describe()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4-Vgg-1zu8Ud",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "validation_targets, validation_examples = parse_labels_and_features(mnist_dataframe[7500:10000])\n",
        "validation_examples.describe()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "wrnAI1v6u8Uh",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Show a random example and its corresponding label."
      ]
    },
    {
      "metadata": {
        "id": "s-euVJVtu8Ui",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "rand_example = np.random.choice(training_examples.index)\n",
        "_, ax = plt.subplots()\n",
        "ax.matshow(training_examples.loc[rand_example].values.reshape(28, 28))\n",
        "ax.set_title(\"Label: %i\" % training_targets.loc[rand_example])\n",
        "ax.grid(False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ScmYX7xdZMXE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Task 1: Build a Linear Model for MNIST\n",
        "\n",
        "First, let's create a baseline model to compare against. The `LinearClassifier` provides a set of *k* one-vs-all classifiers, one for each of the *k* classes.\n",
        "\n",
        "You'll notice that in addition to reporting accuracy, and plotting Log Loss over time, we also display a [**confusion matrix**](https://en.wikipedia.org/wiki/Confusion_matrix).  The confusion matrix shows which classes were misclassified as other classes. Which digits get confused for each other?\n",
        "\n",
        "Also note that we track the model's error using the `log_loss` function. This should not be confused with the loss function internal to `LinearClassifier` that is used for training."
      ]
    },
    {
      "metadata": {
        "id": "cpoVC4TSdw5Z",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def construct_feature_columns():\n",
        "  \"\"\"Construct the TensorFlow Feature Columns.\n",
        "\n",
        "  Returns:\n",
        "    A set of feature columns\n",
        "  \"\"\" \n",
        "  \n",
        "  # There are 784 pixels in each image.\n",
        "  return set([tf.feature_column.numeric_column('pixels', shape=784)])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "kMmL89yGeTfz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Here, we'll make separate input functions for training and for prediction. We'll nest them in `create_training_input_fn()` and `create_predict_input_fn()`, respectively, so we can invoke these functions to return the corresponding `_input_fn`s to pass to our `.train()` and `.predict()` calls."
      ]
    },
    {
      "metadata": {
        "id": "OeS47Bmn5Ms2",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def create_training_input_fn(features, labels, batch_size, num_epochs=None, shuffle=True):\n",
        "  \"\"\"A custom input_fn for sending MNIST data to the estimator for training.\n",
        "\n",
        "  Args:\n",
        "    features: The training features.\n",
        "    labels: The training labels.\n",
        "    batch_size: Batch size to use during training.\n",
        "\n",
        "  Returns:\n",
        "    A function that returns batches of training features and labels during\n",
        "    training.\n",
        "  \"\"\"\n",
        "  def _input_fn(num_epochs=None, shuffle=True):\n",
        "    # Input pipelines are reset with each call to .train(). To ensure model\n",
        "    # gets a good sampling of data, even when number of steps is small, we \n",
        "    # shuffle all the data before creating the Dataset object\n",
        "    idx = np.random.permutation(features.index)\n",
        "    raw_features = {\"pixels\":features.reindex(idx)}\n",
        "    raw_targets = np.array(labels[idx])\n",
        "   \n",
        "    ds = Dataset.from_tensor_slices((raw_features,raw_targets)) # warning: 2GB limit\n",
        "    ds = ds.batch(batch_size).repeat(num_epochs)\n",
        "    \n",
        "    if shuffle:\n",
        "      ds = ds.shuffle(10000)\n",
        "    \n",
        "    # Return the next batch of data.\n",
        "    feature_batch, label_batch = ds.make_one_shot_iterator().get_next()\n",
        "    return feature_batch, label_batch\n",
        "\n",
        "  return _input_fn"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8zoGWAoohrwS",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def create_predict_input_fn(features, labels, batch_size):\n",
        "  \"\"\"A custom input_fn for sending mnist data to the estimator for predictions.\n",
        "\n",
        "  Args:\n",
        "    features: The features to base predictions on.\n",
        "    labels: The labels of the prediction examples.\n",
        "\n",
        "  Returns:\n",
        "    A function that returns features and labels for predictions.\n",
        "  \"\"\"\n",
        "  def _input_fn():\n",
        "    raw_features = {\"pixels\": features.values}\n",
        "    raw_targets = np.array(labels)\n",
        "    \n",
        "    ds = Dataset.from_tensor_slices((raw_features, raw_targets)) # warning: 2GB limit\n",
        "    ds = ds.batch(batch_size)\n",
        "    \n",
        "        \n",
        "    # Return the next batch of data.\n",
        "    feature_batch, label_batch = ds.make_one_shot_iterator().get_next()\n",
        "    return feature_batch, label_batch\n",
        "\n",
        "  return _input_fn"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "G6DjSLZMu8Um",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def train_linear_classification_model(\n",
        "    learning_rate,\n",
        "    steps,\n",
        "    batch_size,\n",
        "    training_examples,\n",
        "    training_targets,\n",
        "    validation_examples,\n",
        "    validation_targets):\n",
        "  \"\"\"Trains a linear classification model for the MNIST digits dataset.\n",
        "  \n",
        "  In addition to training, this function also prints training progress information,\n",
        "  a plot of the training and validation loss over time, and a confusion\n",
        "  matrix.\n",
        "  \n",
        "  Args:\n",
        "    learning_rate: An `int`, the learning rate to use.\n",
        "    steps: A non-zero `int`, the total number of training steps. A training step\n",
        "      consists of a forward and backward pass using a single batch.\n",
        "    batch_size: A non-zero `int`, the batch size.\n",
        "    training_examples: A `DataFrame` containing the training features.\n",
        "    training_targets: A `DataFrame` containing the training labels.\n",
        "    validation_examples: A `DataFrame` containing the validation features.\n",
        "    validation_targets: A `DataFrame` containing the validation labels.\n",
        "      \n",
        "  Returns:\n",
        "    The trained `LinearClassifier` object.\n",
        "  \"\"\"\n",
        "\n",
        "  periods = 10\n",
        "\n",
        "  steps_per_period = steps / periods  \n",
        "  # Create the input functions.\n",
        "  predict_training_input_fn = create_predict_input_fn(\n",
        "    training_examples, training_targets, batch_size)\n",
        "  predict_validation_input_fn = create_predict_input_fn(\n",
        "    validation_examples, validation_targets, batch_size)\n",
        "  training_input_fn = create_training_input_fn(\n",
        "    training_examples, training_targets, batch_size)\n",
        "  \n",
        "  # Create a LinearClassifier object.\n",
        "  my_optimizer = tf.train.AdagradOptimizer(learning_rate=learning_rate)\n",
        "  my_optimizer = tf.contrib.estimator.clip_gradients_by_norm(my_optimizer, 5.0)\n",
        "  classifier = tf.estimator.LinearClassifier(\n",
        "      feature_columns=construct_feature_columns(),\n",
        "      n_classes=10,\n",
        "      optimizer=my_optimizer,\n",
        "      config=tf.estimator.RunConfig(keep_checkpoint_max=1)\n",
        "  )\n",
        "\n",
        "  # Train the model, but do so inside a loop so that we can periodically assess\n",
        "  # loss metrics.\n",
        "  print(\"Training model...\")\n",
        "  print(\"LogLoss error (on validation data):\")\n",
        "  training_errors = []\n",
        "  validation_errors = []\n",
        "  for period in range (0, periods):\n",
        "    # Train the model, starting from the prior state.\n",
        "    classifier.train(\n",
        "        input_fn=training_input_fn,\n",
        "        steps=steps_per_period\n",
        "    )\n",
        "  \n",
        "    # Take a break and compute probabilities.\n",
        "    training_predictions = list(classifier.predict(input_fn=predict_training_input_fn))\n",
        "    training_probabilities = np.array([item['probabilities'] for item in training_predictions])\n",
        "    training_pred_class_id = np.array([item['class_ids'][0] for item in training_predictions])\n",
        "    training_pred_one_hot = tf.keras.utils.to_categorical(training_pred_class_id,10)\n",
        "        \n",
        "    validation_predictions = list(classifier.predict(input_fn=predict_validation_input_fn))\n",
        "    validation_probabilities = np.array([item['probabilities'] for item in validation_predictions])    \n",
        "    validation_pred_class_id = np.array([item['class_ids'][0] for item in validation_predictions])\n",
        "    validation_pred_one_hot = tf.keras.utils.to_categorical(validation_pred_class_id,10)    \n",
        "    \n",
        "    # Compute training and validation errors.\n",
        "    training_log_loss = metrics.log_loss(training_targets, training_pred_one_hot)\n",
        "    validation_log_loss = metrics.log_loss(validation_targets, validation_pred_one_hot)\n",
        "    # Occasionally print the current loss.\n",
        "    print(\"  period %02d : %0.2f\" % (period, validation_log_loss))\n",
        "    # Add the loss metrics from this period to our list.\n",
        "    training_errors.append(training_log_loss)\n",
        "    validation_errors.append(validation_log_loss)\n",
        "  print(\"Model training finished.\")\n",
        "  # Remove event files to save disk space.\n",
        "  _ = map(os.remove, glob.glob(os.path.join(classifier.model_dir, 'events.out.tfevents*')))\n",
        "  \n",
        "  # Calculate final predictions (not probabilities, as above).\n",
        "  final_predictions = classifier.predict(input_fn=predict_validation_input_fn)\n",
        "  final_predictions = np.array([item['class_ids'][0] for item in final_predictions])\n",
        "  \n",
        "  \n",
        "  accuracy = metrics.accuracy_score(validation_targets, final_predictions)\n",
        "  print(\"Final accuracy (on validation data): %0.2f\" % accuracy)\n",
        "\n",
        "  # Output a graph of loss metrics over periods.\n",
        "  plt.ylabel(\"LogLoss\")\n",
        "  plt.xlabel(\"Periods\")\n",
        "  plt.title(\"LogLoss vs. Periods\")\n",
        "  plt.plot(training_errors, label=\"training\")\n",
        "  plt.plot(validation_errors, label=\"validation\")\n",
        "  plt.legend()\n",
        "  plt.show()\n",
        "  \n",
        "  # Output a plot of the confusion matrix.\n",
        "  cm = metrics.confusion_matrix(validation_targets, final_predictions)\n",
        "  # Normalize the confusion matrix by row (i.e by the number of samples\n",
        "  # in each class).\n",
        "  cm_normalized = cm.astype(\"float\") / cm.sum(axis=1)[:, np.newaxis]\n",
        "  ax = sns.heatmap(cm_normalized, cmap=\"bone_r\")\n",
        "  ax.set_aspect(1)\n",
        "  plt.title(\"Confusion matrix\")\n",
        "  plt.ylabel(\"True label\")\n",
        "  plt.xlabel(\"Predicted label\")\n",
        "  plt.show()\n",
        "\n",
        "  return classifier"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ItHIUyv2u8Ur",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Spend 5 minutes seeing how well you can do on accuracy with a linear model of this form. For this exercise, limit yourself to experimenting with the hyperparameters for batch size, learning rate and steps.**\n",
        "\n",
        "Stop if you get anything above about 0.9 accuracy."
      ]
    },
    {
      "metadata": {
        "id": "yaiIhIQqu8Uv",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "classifier = train_linear_classification_model(\n",
        "             learning_rate=0.02,\n",
        "             steps=100,\n",
        "             batch_size=10,\n",
        "             training_examples=training_examples,\n",
        "             training_targets=training_targets,\n",
        "             validation_examples=validation_examples,\n",
        "             validation_targets=validation_targets)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "266KQvZoMxMv",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Solution\n",
        "\n",
        "Click below for one possible solution."
      ]
    },
    {
      "metadata": {
        "id": "lRWcn24DM3qa",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Here is a set of parameters that should attain roughly 0.9 accuracy."
      ]
    },
    {
      "metadata": {
        "id": "TGlBMrUoM1K_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "_ = train_linear_classification_model(\n",
        "    learning_rate=0.03,\n",
        "    steps=1000,\n",
        "    batch_size=30,\n",
        "    training_examples=training_examples,\n",
        "    training_targets=training_targets,\n",
        "    validation_examples=validation_examples,\n",
        "    validation_targets=validation_targets)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "mk095OfpPdOx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Task 2: Replace the Linear Classifier with a Neural Network\n",
        "\n",
        "**Replace the LinearClassifier above with a [`DNNClassifier`](https://www.tensorflow.org/api_docs/python/tf/estimator/DNNClassifier) and find a parameter combination that gives 0.95 or better accuracy.**\n",
        "\n",
        "You may wish to experiment with additional regularization methods, such as dropout. These additional regularization methods are documented in the comments for the `DNNClassifier` class."
      ]
    },
    {
      "metadata": {
        "id": "rm8P_Ttwu8U4",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#\n",
        "# YOUR CODE HERE: Replace the linear classifier with a neural network.\n",
        "#"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "TOfmiSvqu8U9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Once you have a good model, double check that you didn't overfit the validation set by evaluating on the test data that we'll load below.\n"
      ]
    },
    {
      "metadata": {
        "id": "evlB5ubzu8VJ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "mnist_test_dataframe = pd.read_csv(\n",
        "  \"https://dl.google.com/mlcc/mledu-datasets/mnist_test.csv\",\n",
        "  sep=\",\",\n",
        "  header=None)\n",
        "\n",
        "test_targets, test_examples = parse_labels_and_features(mnist_test_dataframe)\n",
        "test_examples.describe()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PDuLd2Hcu8VL",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#\n",
        "# YOUR CODE HERE: Calculate accuracy on the test set.\n",
        "#"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "6sfw3LH0Oycm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Solution\n",
        "\n",
        "Click below for a possible solution."
      ]
    },
    {
      "metadata": {
        "id": "XatDGFKEO374",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The code below is almost identical to the original `LinearClassifer` training code, with the exception of the NN-specific configuration, such as the hyperparameter for hidden units."
      ]
    },
    {
      "metadata": {
        "id": "kdNTx8jkPQUx",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def train_nn_classification_model(\n",
        "    learning_rate,\n",
        "    steps,\n",
        "    batch_size,\n",
        "    hidden_units,\n",
        "    training_examples,\n",
        "    training_targets,\n",
        "    validation_examples,\n",
        "    validation_targets):\n",
        "  \"\"\"Trains a neural network classification model for the MNIST digits dataset.\n",
        "  \n",
        "  In addition to training, this function also prints training progress information,\n",
        "  a plot of the training and validation loss over time, as well as a confusion\n",
        "  matrix.\n",
        "  \n",
        "  Args:\n",
        "    learning_rate: An `int`, the learning rate to use.\n",
        "    steps: A non-zero `int`, the total number of training steps. A training step\n",
        "      consists of a forward and backward pass using a single batch.\n",
        "    batch_size: A non-zero `int`, the batch size.\n",
        "    hidden_units: A `list` of int values, specifying the number of neurons in each layer.\n",
        "    training_examples: A `DataFrame` containing the training features.\n",
        "    training_targets: A `DataFrame` containing the training labels.\n",
        "    validation_examples: A `DataFrame` containing the validation features.\n",
        "    validation_targets: A `DataFrame` containing the validation labels.\n",
        "      \n",
        "  Returns:\n",
        "    The trained `DNNClassifier` object.\n",
        "  \"\"\"\n",
        "\n",
        "  periods = 10\n",
        "  # Caution: input pipelines are reset with each call to train. \n",
        "  # If the number of steps is small, your model may never see most of the data.  \n",
        "  # So with multiple `.train` calls like this you may want to control the length \n",
        "  # of training with num_epochs passed to the input_fn. Or, you can do a really-big shuffle, \n",
        "  # or since it's in-memory data, shuffle all the data in the `input_fn`.\n",
        "  steps_per_period = steps / periods  \n",
        "  # Create the input functions.\n",
        "  predict_training_input_fn = create_predict_input_fn(\n",
        "    training_examples, training_targets, batch_size)\n",
        "  predict_validation_input_fn = create_predict_input_fn(\n",
        "    validation_examples, validation_targets, batch_size)\n",
        "  training_input_fn = create_training_input_fn(\n",
        "    training_examples, training_targets, batch_size)\n",
        "  \n",
        "  # Create the input functions.\n",
        "  predict_training_input_fn = create_predict_input_fn(\n",
        "    training_examples, training_targets, batch_size)\n",
        "  predict_validation_input_fn = create_predict_input_fn(\n",
        "    validation_examples, validation_targets, batch_size)\n",
        "  training_input_fn = create_training_input_fn(\n",
        "    training_examples, training_targets, batch_size)\n",
        "  \n",
        "  # Create feature columns.\n",
        "  feature_columns = [tf.feature_column.numeric_column('pixels', shape=784)]\n",
        "\n",
        "  # Create a DNNClassifier object.\n",
        "  my_optimizer = tf.train.AdagradOptimizer(learning_rate=learning_rate)\n",
        "  my_optimizer = tf.contrib.estimator.clip_gradients_by_norm(my_optimizer, 5.0)\n",
        "  classifier = tf.estimator.DNNClassifier(\n",
        "      feature_columns=feature_columns,\n",
        "      n_classes=10,\n",
        "      hidden_units=hidden_units,\n",
        "      optimizer=my_optimizer,\n",
        "      config=tf.contrib.learn.RunConfig(keep_checkpoint_max=1)\n",
        "  )\n",
        "\n",
        "  # Train the model, but do so inside a loop so that we can periodically assess\n",
        "  # loss metrics.\n",
        "  print(\"Training model...\")\n",
        "  print(\"LogLoss error (on validation data):\")\n",
        "  training_errors = []\n",
        "  validation_errors = []\n",
        "  for period in range (0, periods):\n",
        "    # Train the model, starting from the prior state.\n",
        "    classifier.train(\n",
        "        input_fn=training_input_fn,\n",
        "        steps=steps_per_period\n",
        "    )\n",
        "  \n",
        "    # Take a break and compute probabilities.\n",
        "    training_predictions = list(classifier.predict(input_fn=predict_training_input_fn))\n",
        "    training_probabilities = np.array([item['probabilities'] for item in training_predictions])\n",
        "    training_pred_class_id = np.array([item['class_ids'][0] for item in training_predictions])\n",
        "    training_pred_one_hot = tf.keras.utils.to_categorical(training_pred_class_id,10)\n",
        "        \n",
        "    validation_predictions = list(classifier.predict(input_fn=predict_validation_input_fn))\n",
        "    validation_probabilities = np.array([item['probabilities'] for item in validation_predictions])    \n",
        "    validation_pred_class_id = np.array([item['class_ids'][0] for item in validation_predictions])\n",
        "    validation_pred_one_hot = tf.keras.utils.to_categorical(validation_pred_class_id,10)    \n",
        "    \n",
        "    # Compute training and validation errors.\n",
        "    training_log_loss = metrics.log_loss(training_targets, training_pred_one_hot)\n",
        "    validation_log_loss = metrics.log_loss(validation_targets, validation_pred_one_hot)\n",
        "    # Occasionally print the current loss.\n",
        "    print(\"  period %02d : %0.2f\" % (period, validation_log_loss))\n",
        "    # Add the loss metrics from this period to our list.\n",
        "    training_errors.append(training_log_loss)\n",
        "    validation_errors.append(validation_log_loss)\n",
        "  print(\"Model training finished.\")\n",
        "  # Remove event files to save disk space.\n",
        "  _ = map(os.remove, glob.glob(os.path.join(classifier.model_dir, 'events.out.tfevents*')))\n",
        "  \n",
        "  # Calculate final predictions (not probabilities, as above).\n",
        "  final_predictions = classifier.predict(input_fn=predict_validation_input_fn)\n",
        "  final_predictions = np.array([item['class_ids'][0] for item in final_predictions])\n",
        "  \n",
        "  \n",
        "  accuracy = metrics.accuracy_score(validation_targets, final_predictions)\n",
        "  print(\"Final accuracy (on validation data): %0.2f\" % accuracy)\n",
        "\n",
        "  # Output a graph of loss metrics over periods.\n",
        "  plt.ylabel(\"LogLoss\")\n",
        "  plt.xlabel(\"Periods\")\n",
        "  plt.title(\"LogLoss vs. Periods\")\n",
        "  plt.plot(training_errors, label=\"training\")\n",
        "  plt.plot(validation_errors, label=\"validation\")\n",
        "  plt.legend()\n",
        "  plt.show()\n",
        "  \n",
        "  # Output a plot of the confusion matrix.\n",
        "  cm = metrics.confusion_matrix(validation_targets, final_predictions)\n",
        "  # Normalize the confusion matrix by row (i.e by the number of samples\n",
        "  # in each class).\n",
        "  cm_normalized = cm.astype(\"float\") / cm.sum(axis=1)[:, np.newaxis]\n",
        "  ax = sns.heatmap(cm_normalized, cmap=\"bone_r\")\n",
        "  ax.set_aspect(1)\n",
        "  plt.title(\"Confusion matrix\")\n",
        "  plt.ylabel(\"True label\")\n",
        "  plt.xlabel(\"Predicted label\")\n",
        "  plt.show()\n",
        "\n",
        "  return classifier"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ZfzsTYGPPU8I",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "classifier = train_nn_classification_model(\n",
        "    learning_rate=0.05,\n",
        "    steps=1000,\n",
        "    batch_size=30,\n",
        "    hidden_units=[100, 100],\n",
        "    training_examples=training_examples,\n",
        "    training_targets=training_targets,\n",
        "    validation_examples=validation_examples,\n",
        "    validation_targets=validation_targets)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qXvrOgtUR-zD",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Next, we verify the accuracy on the test set."
      ]
    },
    {
      "metadata": {
        "id": "scQNpDePSFjt",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "mnist_test_dataframe = pd.read_csv(\n",
        "  \"https://dl.google.com/mlcc/mledu-datasets/mnist_test.csv\",\n",
        "  sep=\",\",\n",
        "  header=None)\n",
        "\n",
        "test_targets, test_examples = parse_labels_and_features(mnist_test_dataframe)\n",
        "test_examples.describe()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "EVaWpWKvSHmu",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "predict_test_input_fn = create_predict_input_fn(\n",
        "    test_examples, test_targets, batch_size=100)\n",
        "\n",
        "test_predictions = classifier.predict(input_fn=predict_test_input_fn)\n",
        "test_predictions = np.array([item['class_ids'][0] for item in test_predictions])\n",
        "  \n",
        "accuracy = metrics.accuracy_score(test_targets, test_predictions)\n",
        "print(\"Accuracy on test data: %0.2f\" % accuracy)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "WX2mQBAEcisO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Task 3: Visualize the weights of the first hidden layer.\n",
        "\n",
        "Let's take a few minutes to dig into our neural network and see what it has learned by accessing the `weights_` attribute of our model.\n",
        "\n",
        "The input layer of our model has `784` weights corresponding to the `28×28` pixel input images. The first hidden layer will have `784×N` weights where `N` is the number of nodes in that layer. We can turn those weights back into `28×28` images by *reshaping* each of the `N` `1×784` arrays of weights into `N` arrays of size `28×28`.\n",
        "\n",
        "Run the following cell to plot the weights. Note that this cell requires that a `DNNClassifier` called \"classifier\" has already been trained."
      ]
    },
    {
      "metadata": {
        "id": "eUC0Z8nbafgG",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "print(classifier.get_variable_names())\n",
        "\n",
        "weights0 = classifier.get_variable_value(\"dnn/hiddenlayer_0/kernel\")\n",
        "\n",
        "print(\"weights0 shape:\", weights0.shape)\n",
        "\n",
        "num_nodes = weights0.shape[1]\n",
        "num_rows = int(math.ceil(num_nodes / 10.0))\n",
        "fig, axes = plt.subplots(num_rows, 10, figsize=(20, 2 * num_rows))\n",
        "for coef, ax in zip(weights0.T, axes.ravel()):\n",
        "    # Weights in coef is reshaped from 1x784 to 28x28.\n",
        "    ax.matshow(coef.reshape(28, 28), cmap=plt.cm.pink)\n",
        "    ax.set_xticks(())\n",
        "    ax.set_yticks(())\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "kL8MEhNgrx9N",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The first hidden layer of the neural network should be modeling some pretty low level features, so visualizing the weights will probably just show some fuzzy blobs or possibly a few parts of digits.  You may also see some neurons that are essentially noise -- these are either unconverged or they are being ignored by higher layers.\n",
        "\n",
        "It can be interesting to stop training at different numbers of iterations and see the effect.\n",
        "\n",
        "**Train the classifier for 10, 100 and respectively 1000 steps. Then run this visualization again.**\n",
        "\n",
        "What differences do you see visually for the different levels of convergence?"
      ]
    }
  ]
}