2-fashion-mnist-with-keras-eager-and-tf-data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2-B5UZxUchfd"
   },
   "source": [
    "# Fashion-MNIST with tf.keras and tf.data\n",
    "\n",
    "In this notebook, you'll learn how create a `tf.data` [Dataset](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) for the Fashion-MNIST dataset. The `tf.data` API provides tools for working with data (including common functionality, like shuffling, batching, etc), as well as high performance utilities (like parallel reads, and pre-fetching to GPUs) - should you need these down the road. Here, we'll wrap Fashion-MNIST with `tf.data`, just to introduce the API. Later, you'll learn how to use `tf.data` on your own data, say, to read a directory of images from disk or S3 bucket.\n",
    "\n",
    "To learn more about `tf.data`, you can check out [this](https://www.youtube.com/watch?v=EHHdyM3NNiA&index=26&list=PLQY2H8rRoyvxjVx3zfw4vA4cvlKogyLNN) talk from the [2018 TensorFlow Developer Summit](https://www.tensorflow.org/dev-summit/).\n",
    "\n",
    "The rest of the code similar to the previous notebook, with three changes:\n",
    "\n",
    "1. Instead of using `model.fit`, we use `model.train_on_batch` as we iterate over our dataset. \n",
    "2. We will create a TensorFlow optimizer to pass to the Keras model.\n",
    "\n",
    "### Eager execution\n",
    "Eager execution is a mode for running TensorFlow that works just like regular Python. To learn more about eager, check out [this talk](https://www.youtube.com/watch?v=T8AW0fKP0Hs&list=PLQY2H8rRoyvxjVx3zfw4vA4cvlKogyLNN&index=8) from the same summit. In short Eager execution  is an imperative programming environment that evaluates operations immediately. TensorFlow 2 runs with eager execution by default, which is supported by `tf.keras`, and this is super useful for inspecting your program and debugging it. All the `tf.keras` model-building APIs are compatible with eager execution!\n",
    "\n",
    "So to get started let's create our model from the last notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "902Rjd5DZroO"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "evWlYUkYefG8"
   },
   "outputs": [],
   "source": [
    "(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.fashion_mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "JA3braIeeggc"
   },
   "outputs": [],
   "source": [
    "TRAINING_SIZE = len(train_images)\n",
    "TEST_SIZE = len(test_images)\n",
    "\n",
    "# Reshape from (N, 28, 28) to (N, 28*28=784)\n",
    "train_images = np.reshape(train_images, (TRAINING_SIZE, 784))\n",
    "test_images = np.reshape(test_images, (TEST_SIZE, 784))\n",
    "\n",
    "# Convert the array to float32 as opposed to uint8\n",
    "train_images = train_images.astype(np.float32)\n",
    "test_images = test_images.astype(np.float32)\n",
    "\n",
    "# Convert the pixel values from integers between 0 and 255 to floats between 0 and 1\n",
    "train_images /= 255\n",
    "test_images /=  255"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "wbwRqi0BeicT"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before 9\n",
      "After [0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]\n"
     ]
    }
   ],
   "source": [
    "NUM_CAT = 10\n",
    "\n",
    "print(\"Before\", train_labels[0]) # The format of the labels before conversion\n",
    "\n",
    "train_labels  = tf.keras.utils.to_categorical(train_labels, NUM_CAT)\n",
    "\n",
    "print(\"After\", train_labels[0]) # The format of the labels after conversion\n",
    "\n",
    "test_labels = tf.keras.utils.to_categorical(test_labels, NUM_CAT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "XFtvnaI5jU_5"
   },
   "outputs": [],
   "source": [
    "# Cast the labels to floats, needed later\n",
    "train_labels = train_labels.astype(np.float32)\n",
    "test_labels = test_labels.astype(np.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "jjNr3gr3ejh2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 512)               401920    \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 10)                5130      \n",
      "=================================================================\n",
      "Total params: 407,050\n",
      "Trainable params: 407,050\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = tf.keras.Sequential()\n",
    "model.add(tf.keras.layers.Dense(512, activation=tf.nn.relu, input_shape=(784,)))\n",
    "model.add(tf.keras.layers.Dense(NUM_CAT, activation=tf.nn.softmax))\n",
    "\n",
    "\n",
    "# Create a Keras version optimizer\n",
    "optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001)\n",
    "\n",
    "# We will now compile and print out a summary of our model\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=optimizer,\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zUZPvDQYe5Xu"
   },
   "source": [
    "### Step 1) Create a tf.data Dataset\n",
    "\n",
    "Here, we'll use the `tf.data.Dataset` [API](https://www.tensorflow.org/api_docs/python/tf/data) to convert the Numpy arrays into a TensorFlow dataset.\n",
    "\n",
    "Next, we will create a simple `for` loop that will serve as our introduction into creating custom training loops. Although this essentially does the same thing as `model.fit` it allows us to get creative and customize the overall training process (should you like to, if you venture into research) and collect different metrics throughout the process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "sdBd2pd_fdue"
   },
   "outputs": [],
   "source": [
    "BATCH_SIZE=128\n",
    "\n",
    "# Because tf.data may work with potentially **large** collections of data\n",
    "# we do not shuffle the entire dataset by default\n",
    "# Instead, we maintain a buffer of SHUFFLE_SIZE elements\n",
    "# and sample from there.\n",
    "SHUFFLE_SIZE = 10000 \n",
    "\n",
    "# Create the dataset\n",
    "dataset = tf.data.Dataset.from_tensor_slices((train_images, train_labels))\n",
    "dataset = dataset.shuffle(SHUFFLE_SIZE)\n",
    "dataset = dataset.batch(BATCH_SIZE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ksAR-C6xgUu4"
   },
   "source": [
    "### Step 2) Iterate over the dataset\n",
    "Here, we'll iterate over the dataset, and train our model using `model.train_on_batch`. To learn more about the elements returned from the dataset, you can print them out and try the `.numpy()` method.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "kNgnUKPvgSCz"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch #1\t Loss: 0.496344\tAccuracy: 0.802083\n",
      "Epoch #2\t Loss: 0.427685\tAccuracy: 0.822917\n",
      "Epoch #3\t Loss: 0.380092\tAccuracy: 0.833333\n",
      "Epoch #4\t Loss: 0.365240\tAccuracy: 0.843750\n",
      "Epoch #5\t Loss: 0.300417\tAccuracy: 0.864583\n"
     ]
    }
   ],
   "source": [
    "EPOCHS=5\n",
    "\n",
    "for epoch in range(EPOCHS):\n",
    "    for images, labels in dataset:\n",
    "        train_loss, train_accuracy = model.train_on_batch(images, labels)\n",
    "  \n",
    "    # Here you can gather any metrics or adjust your training parameters\n",
    "    print('Epoch #%d\\t Loss: %.6f\\tAccuracy: %.6f' % (epoch + 1, train_loss, train_accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "tg5U3Iqkgo3J"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000==============================] - 0s 48us/sample - loss: 0.3557 - acc: 0.8688\n",
      "Test accuracy: 0.87\n"
     ]
    }
   ],
   "source": [
    "loss, accuracy = model.evaluate(test_images, test_labels)\n",
    "print('Test accuracy: %.2f' % (accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Fu0eSNa9kEFZ"
   },
   "source": [
    "### Congratulations\n",
    "You have trained a model on Fashion-MNIST using Keras and `tf.data`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "default_view": {},
   "name": "2-mnist-with-keras-eager-and-tf-data.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}