1 - Getting Started and Data Prep.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1 - Getting Started and Data Preparation\n",
    "\n",
    "In this deep learning tutorial notebook, we will cover the following topics:\n",
    "\n",
    "* Hardware and software tools for deep learning\n",
    "* Working with Keras and Tensorflow\n",
    "* Loading and inspecting the CIFAR10 data\n",
    "\n",
    "Note about shell commands in Jupyter Notebooks: To avoid having to open a separate terminal window, we will be using a Jupyter feature that allows you to execute commands in the bash shell by starting the line with a `!`.  For example:\n",
    "```\n",
    "! ls *.ipynb\n",
    "```\n",
    "will show all the Jupyter notebooks in the current directory.  These commands could just as well be typed in a terminal window."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tools for Deep Learning\n",
    "\n",
    "In this tutorial, we will focus on the [Keras](https://keras.io) neural networking library.  Keras provides an easy-to-use, high level Python API for building and training neural networks, and has quickly become the most popular interface for deep learning.  Part of the power of Keras is that it delegates the computation to one of the following deep learning frameworks:\n",
    "* TensorFlow\n",
    "* Theano\n",
    "* Microsoft Cognitive Toolkit (also known as \"CNTK\")\n",
    "\n",
    "TensorFlow is the default Keras backend, and that is what we will use for this tutorial.  This will enable us to make use of TensorFlow tools, like TensorBoard, in later tutorial units.\n",
    "\n",
    "### Hardware/OS Requirements\n",
    "\n",
    "The benefit of libraries like Keras/TensorFlow is that they are designed to be *hardware indepenedent*.  The same code model code (with few exceptions) will execute on either the CPU or the GPU.  This allows you to train on a system with a GPU, and deploy on a server with only CPUs, or any combination.\n",
    "\n",
    "We will be using GPUs to speed up our training today.  Any NVIDIA GPU from the last 5 years can be used with TensorFlow, although for most problems we suggest a GPU with at least 8 GB of memory.  The GeForce product line is fine for learning, but generally the Titan or Quadro cards are better for workstations and the Tesla cards for servers.  For deep learning, we generally suggest:\n",
    "\n",
    "* GeForce 1080, 1080 Ti, 2080, 2080 Ti\n",
    "* Titan Xp, V, RTX\n",
    "* Quadro P100 or V100\n",
    "* Tesla K40, K80, P100, or V100\n",
    "\n",
    "The Tesla K80, P100 and V100 can be rented from most cloud computing providers.\n",
    "\n",
    "For the best experience, we suggest using Linux for GPU-accelerated deep learning.  We will be using Ubuntu Linux 64-bit 18.04 in this tutorial, but any Linux distribution supported by the [official NVIDIA drivers](http://www.nvidia.com/Download/index.aspx).  If you are using Anaconda, you do not need to install CUDA to use the GPU-accelerated packages, but the official NVIDIA drivers (not the free Nouveau drivers that come preinstalled with some Linux distributions) are required.\n",
    "\n",
    "On Linux, we can check the GPU hardware available on our system with the `nvidia-smi` command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This table shows the GPU name, total and used GPU memory, temperature, power consumption, and utilization.  For this tutorial, we will be using the slightly older Tesla P100 cards with 16 GB of memory per device."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Software Requirements\n",
    "\n",
    "Once you have access to a suitable NVIDIA GPU and operating system drivers, you can get started very quickly by installing [Anaconda Individual Edition](https://www.anaconda.com/products/individual).\n",
    "\n",
    "Once Anaconda is installed, you can use `conda` to install the GPU-accelerated Keras/TensorFlow:\n",
    "```\n",
    "conda install keras-gpu\n",
    "```\n",
    "(For this tutorial, we also use the `notebook`, `h5py`, `bokeh`, and `holoviews` packages.)\n",
    "\n",
    "We have already preloaded these packages into the tutorial environment, which you can verify with `conda list`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! conda list 'keras|tensorflow|bokeh|holoviews|notebook|h5py'  # Using a regular expression to only show some packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working With Keras and TensorFlow\n",
    "\n",
    "Now that we have verified the correct conda packages are installed, let's start importing some libraries.  First, we are going to import Holoviews (for plotting) and enable the Bokeh backend:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "np.warnings.filterwarnings('ignore')  # Hide np.floating warning\n",
    "import holoviews as hv\n",
    "import colorcet as cc\n",
    "hv.extension('bokeh')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's import Keras:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow\n",
    "tensorflow.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TensorFlow and GPU Memory\n",
    "\n",
    "Even though we are using Keras for the majority of the tutorial, we can also import and access TensorFlow directly.\n",
    "\n",
    "Allocating and deallocating GPU memory can be slow (much slower than on the CPU), so most deep learning frameworks use *memory pools* to speed things up.  When a TensorFlow session initializes the GPU, it grabs 90% of the GPU memory in one big block, and then internally divides it up for different arrays.  This is normally not a problem, but we will have several notebooks open at once in this tutorial, each one with its own TensorFlow session.  For that reason, we will use the following trick to tell TensorFlow it is OK to grow its allocation as needed, rather than taking all 90% up front."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "# TF 1\n",
    "# config = tf.compat.v1.ConfigProto()\n",
    "# config.gpu_options.allow_growth=True\n",
    "# sess = tf.compat.v1.Session(config=config)\n",
    "\n",
    "# TF2\n",
    "gpu_devices = tf.config.experimental.list_physical_devices('GPU')\n",
    "for device in gpu_devices:\n",
    "    tf.config.experimental.set_memory_growth(device, True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's verify our GPUs have been detected by TensorFlow:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.python.client import device_lib\n",
    "\n",
    "device_lib.list_local_devices()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see that TensorFlow has detected both CPU and GPU devices."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading and Inspecting the CIFAR10 Data\n",
    "\n",
    "Keras includes a [datasets](https://keras.io/datasets/) package that will automatically download and import standard test datasets into NumPy arrays.\n",
    "\n",
    "For this tutorial, we will use the CIFAR10 image set, which has the following properties:\n",
    "\n",
    "* 32x32 pixel RGB images\n",
    "* Each image is from 1 of 10 categories (represented numerically from 0 through 9)\n",
    "* 50000 training images, 10000 test images\n",
    "\n",
    "Let's load the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import cifar10\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
    "# The data only has numeric categories so we also have the string labels below \n",
    "cifar10_labels = np.array(['airplane', 'automobile', 'bird', 'cat', 'deer', \n",
    "                           'dog', 'frog', 'horse', 'ship', 'truck'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is organized into NumPy arrays of image data and label data, which we can see from the shape information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('x_train shape:', x_train.shape)\n",
    "print('y_train shape:', y_train.shape)\n",
    "print('x_test shape:', x_test.shape)\n",
    "print('y_test shape:', y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the `x` arrays, the indices are ordered:\n",
    "\n",
    "* image number\n",
    "* image row\n",
    "* image column\n",
    "* color channel (0=R, 1=G, 2=B)\n",
    "\n",
    "We can pull out a single image and plot it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For more information on Holoviews configuration, see http://holoviews.org/user_guide/Customizing_Plots.html\n",
    "i = 12\n",
    "print(cifar10_labels[y_train[i][0]])\n",
    "layout = hv.RGB(x_train[i])\n",
    "\n",
    "hv.output(\n",
    "    layout.opts(\n",
    "        hv.opts.RGB(xaxis=None, yaxis=None),\n",
    "    ),\n",
    "    size=48\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i = 12\n",
    "print(cifar10_labels[y_train[i][0]])\n",
    "layout = hv.Image(x_train[i,:,:,0], label='r') \\\n",
    "  + hv.Image(x_train[i,:,:,1], label='g') \\\n",
    "  + hv.Image(x_train[i,:,:,2], label='b')\n",
    "\n",
    "hv.output(\n",
    "    layout.opts(\n",
    "        hv.opts.Image(xaxis=None, yaxis=None, cmap='gray')\n",
    "    ),\n",
    "    size=48\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to see a bunch of images from a particular class, we can use NumPy fancy indexing and Holoviews layouts to make a grid:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Deer are class 4\n",
    "deer = (y_test[:,0] == 4)\n",
    "\n",
    "\n",
    "images = [hv.RGB(x_test[deer][i]) for i in range(24)]\n",
    "hv.output(\n",
    "    hv.Layout(images).cols(8).opts(\n",
    "        hv.opts.RGB(xaxis=None, yaxis=None)\n",
    "    ),\n",
    "    size=32\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that Holoviews/Bokeh/Matplotlib all assume images are in RGB format, but not all systems load images that way.  For example, [OpenCV](https://opencv.org/) loads images from files and webcams in BGR format.  To get a sense what that error looks like, we can simulate what that would look like by reversing the 4th axis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i = 12\n",
    "layout = hv.RGB(x_train[i], label='Correct (RGB)') + hv.RGB(x_train[i,:,:,::-1], label='Flipped (BGR)')\n",
    "\n",
    "hv.output(\n",
    "    layout.opts(\n",
    "        hv.opts.RGB(xaxis=None, yaxis=None)\n",
    "    ),\n",
    "    size=48\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rescaling Data for Neural Networks\n",
    "\n",
    "For reasons of numerical precision and stability, it is a good idea to scale inputs to the network so that they are floating point numbers approximately between -1 and 1.  For best training performance on the GPU, using 32-bit floating point numbers is preferred since those have higher performance (between 2x and 24x depending on your GPU) than 64-bit floating point numbers.\n",
    "\n",
    "We can use standard NumPy functions and expressions to make this conversion:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_norm = x_train.astype('float32')\n",
    "x_test_norm = x_test.astype('float32')\n",
    "x_train_norm /= 255\n",
    "x_test_norm /= 255\n",
    "print('x_train dtype:', x_train_norm .dtype)\n",
    "print('x_train min/max:', x_train_norm .min(), x_train_norm .max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the output labels of the network, we will use one-hot encoding.  To use one-hot encoding, we need to know the number of categories (or \"classes\") in our data.  For CIFAR10, that number is 10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras.utils\n",
    "\n",
    "num_classes = 10\n",
    "\n",
    "y_train_norm = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test_norm = keras.utils.to_categorical(y_test, num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can look at the first 5 rows to see how there is a single `1` in each column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('y_train shape:', y_train_norm.shape)\n",
    "y_train_norm[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that if we ever want to convert one-hot encoding back to category numbers, we can use the NumPy `argmax` function to find the column number with the largest value along axis 1 for each row:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.argmax(y_train_norm, axis=1)[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because these transformations of the input and category labels are quick, we will not load pre-transformed data from disk in future units.  However, if we wanted to save the transformed arrays, we could save them in an HDF5 file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import h5py\n",
    "\n",
    "with h5py.File('transformed_data.hdf5', 'w') as f:\n",
    "    f.create_dataset('x_train', data=x_train_norm)\n",
    "    f.create_dataset('y_train', data=y_train_norm)\n",
    "    f.create_dataset('x_test', data=x_test_norm)\n",
    "    f.create_dataset('y_test', data=y_test_norm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These files can get very big:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! ls -lh transformed_data.hdf5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To load the data later:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with h5py.File('transformed_data.hdf5', 'r') as f:\n",
    "    x_train_norm = f['x_train'][:] # The extra slice [:] forces h5py to load into memory\n",
    "    y_train_norm = f['y_train'][:] # The extra slice [:] forces h5py to load into memory\n",
    "    x_test_norm = f['x_test'][:] # The extra slice [:] forces h5py to load into memory\n",
    "    y_test_norm = f['y_test'][:] # The extra slice [:] forces h5py to load into memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(x_train_norm.shape)\n",
    "print(x_train_norm.dtype)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}