07-Getting-Started-TF2.0.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### New Features in TensorFlow 2.0\n",
    "\n",
    "TensorFlow 2.0 is mostly a marketing move and some cleanup in the TensorFlow API. Nevertheless, whenever you consider doing deep learning and want to deploy a model, you will find yourself using TensorFlow.\n",
    "\n",
    "Let's start off with a simple way to install / upgrade both the CPU and GPU version of TensorFlow in one line of code. This is not default in the popular Google Colab app yet, but it's rumored to arrive soon.\n",
    "\n",
    "`!pip install --upgrade tensorflow-gpu`\n",
    "\n",
    "All of the upcoming code in this article presumes that you have imported the tensorflow package in your Python program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should verify that you are running the correct version, \n",
    "TensorFlow 2.0, by the first line of code. All it does is call `__version__` from TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Your TensorFlow version: 2.0.0\n"
     ]
    }
   ],
   "source": [
    "print(('Your TensorFlow version: {0}').format(tf.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Default Eager Execution\n",
    "\n",
    "Eager execution means that the interpreter executes line by line, making it much better at and faster when debugging. There is also some cleanup in how graphs are made, which makes it fairly simple – in previous TensorFlow versions, you needed to manually make a graph.\n",
    "\n",
    "This is actually huge, because you reduce the training code \n",
    "\n",
    "**from this**.\n",
    "\n",
    "```python\n",
    "with tf.Session() as session:\n",
    "  session.run(tf.global_variables_initializer())\n",
    "  session.run(tf.tables_initializer())\n",
    "  model.fit(X_train, Y_train, \n",
    "            validation_data=(X_val, Y_val),\n",
    "            epochs=50, batch_size=32)\n",
    "```\n",
    "To that.\n",
    "\n",
    "```python\n",
    "model.fit(X_train, Y_train, \n",
    "          validation_data=(X_val, Y_val),\n",
    "          epochs=50, batch_size=32)\n",
    "```\n",
    "\n",
    "There is no need for sessions or any of those TensorFlow variables, this is just regular Python code executing. It's nice.\n",
    "\n",
    "Here is the official word on the new version of TensorFlow with regards to Eager Execution:\n",
    "\n",
    "`TensorFlow 1.X requires users to manually stitch together an abstract syntax tree (the graph) by making tf.* API calls. It then requires users to manually compile the abstract syntax tree by passing a set of output tensors and input tensors to a session.run() call. TensorFlow 2.0 executes eagerly (like Python normally does) and in 2.0, graphs and sessions should feel like implementation details.\n",
    "One notable byproduct of eager execution is that tf.control_dependencies() is no longer required, as all lines of code execute in order (within a tf.function, code with side effects execute in the order written).\n",
    "The new eager execution feature is actually a great move for TensorFlow, as it gets confusing when you can't immediately evaluate your code, just like in all your other Python code.`\n",
    "\n",
    "### Verify Eager Execution and GPU Devices\n",
    "\n",
    "Eager execution is this big new feature, that allows for many things, as explained earlier – but let's just make sure that we are actually running in eager execution mode.\n",
    "\n",
    "And while we are at it, we should check for which devices we want to run our code on – after all, GPUs are way faster than CPUs when it comes to Deep Learning tasks.\n",
    "\n",
    "Eager Execution Check\n",
    "To verify whether you are running eager execution or not, I have made a small if-else statement that will tell you if you are.\n",
    "\n",
    "Are you running eager execution? And how can you turn it off, if you wish to.\n",
    "If you are not running eager execution, then there is a way to manually do it, or you could just try upgrading your TensorFlow version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Eager execution is enabled (running operations immediately)\n",
      "\n",
      "Turn eager execution off by running: \n",
      "from tensorflow.python.framework.ops import disable_eager_execution\n",
      "disable_eager_execution()\n"
     ]
    }
   ],
   "source": [
    "if(tf.executing_eagerly()):\n",
    "    print('Eager execution is enabled (running operations immediately)\\n')\n",
    "    print(('Turn eager execution off by running: \\n{0}\\n{1}').format('' \\\n",
    "        'from tensorflow.python.framework.ops import disable_eager_execution', \\\n",
    "        'disable_eager_execution()'))\n",
    "else:\n",
    "    print('You are not running eager execution. TensorFlow version >= 2.0.0' \\\n",
    "          'has eager execution enabled by default.')\n",
    "    print(('Turn on eager execution by running: \\n\\n{0}\\n\\nOr upgrade '\\\n",
    "           'your tensorflow version by running:\\n\\n{1}').format(\n",
    "           'tf.compat.v1.enable_eager_execution()',\n",
    "           '!pip install --upgrade tensorflow\\n' \\\n",
    "           '!pip install --upgrade tensorflow-gpu'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Specific Devices (GPUs/CPUs)\n",
    "\n",
    "Let's say we are interested in knowing if we have a GPU device available – or if we know there is a GPU in our machine, we can test if TensorFlow recognizes that it exists. If not, then perhaps you should try and reinstall CUDA and cuDNN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is your GPU available for use?\n",
      "No, your GPU is NOT available: False\n",
      "\n",
      "Your devices that are available:\n",
      "['/physical_device:CPU:0', '/physical_device:XLA_CPU:0']\n"
     ]
    }
   ],
   "source": [
    "print(('Is your GPU available for use?\\n{0}').format(\n",
    "    'Yes, your GPU is available: True' if tf.test.is_gpu_available() == True else 'No, your GPU is NOT available: False'\n",
    "))\n",
    "\n",
    "print(('\\nYour devices that are available:\\n{0}').format(\n",
    "    [device.name for device in tf.config.experimental.list_physical_devices()]\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['/device:CPU:0', '/device:XLA_CPU:0']\n"
     ]
    }
   ],
   "source": [
    "# A second method for getting devices:\n",
    "from tensorflow.python.client import device_lib\n",
    "print([device.name for device in device_lib.list_local_devices() if device.name != None])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### On Google Colab\n",
    "\n",
    "My expected output would be that there should at least be a CPU available, and a GPU if you are running it in Google Colab – if no GPU shows up in Google Colab then you need to go to Edit > Notebook Settings > Hardware Accelerator and pick GPU.\n",
    "\n",
    "As expected, we indeed have a CPU and GPU available in Google Colab:\n",
    "\n",
    "Is your GPU available for use?\n",
    "Yes, your GPU is available: True\n",
    "\n",
    "```shell]\n",
    "Your devices that are available:\n",
    "['/physical_device:CPU:0', '/physical_device:XLA_CPU:0', '/physical_device:XLA_GPU:0', '/physical_device:GPU:0']\n",
    "Great, we know we have a GPU available called GPU:0.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But how do we explicitly use it? First, you should know that TensorFlow by default uses your GPU where it can (not every operation can use the GPU).\n",
    "\n",
    "But if you want to be absolute certain that your code is executed on the GPU, here is a code piece comparing time spent using the CPU versus GPU.\n",
    "\n",
    "The simple operation here is creating a constant with tf.constant and an identity matrix with tf.eye, which we will discuss later in the Linear Algebra section.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[1. 0.]\n",
      " [0. 1.]], shape=(2, 2), dtype=float32)\n",
      "0.00177001953125\n"
     ]
    },
    {
     "ename": "RuntimeError",
     "evalue": "/job:localhost/replica:0/task:0/device:GPU:0 unknown device.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-6-d69c41546ca8>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     25\u001b[0m     \u001b[0;31m# Doing operations on CPU\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     26\u001b[0m     \u001b[0mA\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconstant\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 27\u001b[0;31m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0meye\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     28\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     29\u001b[0m     \u001b[0;31m# Printing how long it took with CPU\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/linalg_ops.py\u001b[0m in \u001b[0;36meye\u001b[0;34m(num_rows, num_columns, batch_shape, dtype, name)\u001b[0m\n\u001b[1;32m    167\u001b[0m                              \u001b[0mbatch_shape\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbatch_shape\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    168\u001b[0m                              \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 169\u001b[0;31m                              name=name)\n\u001b[0m\u001b[1;32m    170\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    171\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/linalg_ops_impl.py\u001b[0m in \u001b[0;36meye\u001b[0;34m(num_rows, num_columns, batch_shape, dtype, name)\u001b[0m\n\u001b[1;32m     73\u001b[0m         \u001b[0mshape\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbatch_shape\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mnum_rows\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_columns\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     74\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 75\u001b[0;31m     \u001b[0mdiag_ones\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0marray_ops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mones\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdiag_shape\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     76\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mis_square\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     77\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0marray_ops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmatrix_diag\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdiag_ones\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py\u001b[0m in \u001b[0;36mones\u001b[0;34m(shape, dtype, name)\u001b[0m\n\u001b[1;32m   2569\u001b[0m         \u001b[0;31m# Create a constant if it won't be very big. Otherwise create a fill op\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2570\u001b[0m         \u001b[0;31m# to prevent serialized GraphDefs from becoming too large.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2571\u001b[0;31m         \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_constant_if_small\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mshape\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2572\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0moutput\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2573\u001b[0m           \u001b[0;32mreturn\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py\u001b[0m in \u001b[0;36m_constant_if_small\u001b[0;34m(value, shape, dtype, name)\u001b[0m\n\u001b[1;32m   2305\u001b[0m   \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2306\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprod\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0;36m1000\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2307\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mconstant\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mshape\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2308\u001b[0m   \u001b[0;32mexcept\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2309\u001b[0m     \u001b[0;31m# Happens when shape is a Tensor, list with Tensor elements, etc.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/constant_op.py\u001b[0m in \u001b[0;36mconstant\u001b[0;34m(value, dtype, shape, name)\u001b[0m\n\u001b[1;32m    225\u001b[0m   \"\"\"\n\u001b[1;32m    226\u001b[0m   return _constant_impl(value, dtype, shape, name, verify_shape=False,\n\u001b[0;32m--> 227\u001b[0;31m                         allow_broadcast=True)\n\u001b[0m\u001b[1;32m    228\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    229\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/constant_op.py\u001b[0m in \u001b[0;36m_constant_impl\u001b[0;34m(value, dtype, shape, name, verify_shape, allow_broadcast)\u001b[0m\n\u001b[1;32m    233\u001b[0m   \u001b[0mctx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcontext\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    234\u001b[0m   \u001b[0;32mif\u001b[0m \u001b[0mctx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexecuting_eagerly\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 235\u001b[0;31m     \u001b[0mt\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconvert_to_eager_tensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mctx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    236\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mshape\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    237\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/constant_op.py\u001b[0m in \u001b[0;36mconvert_to_eager_tensor\u001b[0;34m(value, ctx, dtype)\u001b[0m\n\u001b[1;32m     94\u001b[0m       \u001b[0mdtype\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdtypes\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_dtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_datatype_enum\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     95\u001b[0m   \u001b[0mctx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mensure_initialized\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 96\u001b[0;31m   \u001b[0;32mreturn\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mEagerTensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mctx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdevice_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     97\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     98\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mRuntimeError\u001b[0m: /job:localhost/replica:0/task:0/device:GPU:0 unknown device."
     ]
    }
   ],
   "source": [
    "# On my laptop without a GPU\n",
    "import time\n",
    "\n",
    "cpu_slot = 0\n",
    "gpu_slot = 0\n",
    "\n",
    "# Using CPU at slot 0\n",
    "with tf.device('/CPU:' + str(cpu_slot)):\n",
    "    # Starting a timer\n",
    "    start = time.time()\n",
    "\n",
    "    # Doing operations on CPU\n",
    "    A = tf.constant([[3, 2], [5, 2]])\n",
    "    print(tf.eye(2,2))\n",
    "\n",
    "    # Printing how long it took with CPU\n",
    "    end = time.time() - start\n",
    "    print(end)\n",
    "\n",
    "# Using the GPU at slot 0\n",
    "with tf.device('/GPU:' + str(gpu_slot)):\n",
    "    # Starting a timer\n",
    "    start = time.time()\n",
    "\n",
    "    # Doing operations on CPU\n",
    "    A = tf.constant([[3, 2], [5, 2]])\n",
    "    print(tf.eye(2,2))\n",
    "\n",
    "    # Printing how long it took with CPU\n",
    "    end = time.time() - start\n",
    "    print(end)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**On Google Colab or any GPU device however...**\n",
    "\n",
    "```shell\n",
    "Tensor(\"eye/diag:0\", shape=(2, 2), dtype=float32, device=/device:CPU:0)\n",
    "0.011214733123779297\n",
    "Tensor(\"eye_1/diag:0\", shape=(2, 2), dtype=float32, device=/device:GPU:0)\n",
    "0.0019173622131347656\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that how long it takes will vary each time, but the GPU should always outperform in these types of tasks. We could easily imagine how much this would help us with larger computations. In particular, when there is millions/billions of operations executed on a GPU, we do see a significant speed up of neural networks – always use a GPU, if available.\n",
    "\n",
    "## Common Use Operations\n",
    "\n",
    "The bread and butter of TensorFlow, the most commonly used operations. We are going to take a look at the following\n",
    "\n",
    "1. Making tensors with tf.constant and tf.Variable\n",
    "2. Concatenation of two tensors with tf.concat\n",
    "3. Making tensors with tf.zeros or tf.ones\n",
    "4. Reshaping data with tf.reshape\n",
    "5. Casting tensors to other data types with tf.cast\n",
    "\n",
    "### How to make tensors with `tf.contant and tf.Variable`\n",
    "\n",
    "Perhaps one of the simplest operations in tensorflow is making a constant or variable. You simply call the `tf.constant` or `tf.Variable` function and specify an array of arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Making a constant tensor A, that does not change\n",
    "A = tf.constant([[3, 2],\n",
    "                 [5, 2]])\n",
    "\n",
    "# Making a Variable tensor VA, which can change. Notice it's .Variable\n",
    "VA = tf.Variable([[3, 2],\n",
    "                 [5, 2]])\n",
    "\n",
    "# Making another tensor B\n",
    "B = tf.constant([[9, 5],\n",
    "                 [1, 3]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code piece gives us three tensors; the constant A, the variable VA and the constant B.\n",
    "\n",
    "### How to concatenate two tensors with tf.concat\n",
    "Let's say that we have two tensors, perhaps it could be two observations. We want to concat the two tensors A and B into a single variable in Python – how do we do it?\n",
    "\n",
    "We simply use the `tf.concat`, and specify the values and axis.\n",
    "\n",
    "**NOTE** The first output will be concatenating column-wise by axis=1 and the second will be concatenating row-wise by axis=0 – meaning we add the data either rightwards (columns) or downwards (rows)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Adding B's columns to A:\n",
      "[[3 2 9 5]\n",
      " [5 2 1 3]]\n",
      "\n",
      "Adding B's rows to A:\n",
      "[[3 2]\n",
      " [5 2]\n",
      " [9 5]\n",
      " [1 3]]\n"
     ]
    }
   ],
   "source": [
    "# Making a constant tensor A, that does not change\n",
    "A = tf.constant([[3, 2],\n",
    "                 [5, 2]])\n",
    "\n",
    "# Making a Variable tensor VA, which can change. Notice it's .Variable\n",
    "VA = tf.Variable([[3, 2],\n",
    "                 [5, 2]])\n",
    "\n",
    "# Making another tensor B\n",
    "B = tf.constant([[9, 5],\n",
    "                 [1, 3]])\n",
    "\n",
    "# Concatenate columns\n",
    "AB_concatenated = tf.concat(values=[A, B], axis=1)\n",
    "print(('Adding B\\'s columns to A:\\n{0}').format(\n",
    "    AB_concatenated.numpy()\n",
    "))\n",
    "\n",
    "# Concatenate rows\n",
    "AB_concatenated = tf.concat(values=[A, B], axis=0)\n",
    "print(('\\nAdding B\\'s rows to A:\\n{0}').format(\n",
    "    AB_concatenated.numpy()\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to make tensors with tf.zeros and tf.ones\n",
    "\n",
    "Creating tensors with just tf.constant and tf.Variable can be tedious if you want to create big tensors. Imagine you want to create random noise – well, you could do that by making a tensor with tf.zeros or tf.ones.\n",
    "\n",
    "All we need to specify is the shape in the format `shape=[rows, columns]` and a dtype, if it matters at all. The number of rows and columns are arbitrary, and you could in principle create 4K images (as noise)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor full of zeros as int32, 3 rows and 4 columns:\n",
      "[[0 0 0 0]\n",
      " [0 0 0 0]\n",
      " [0 0 0 0]]\n",
      "\n",
      "Tensor full of ones as float32, 5 rows and 3 columns:\n",
      "[[1. 1. 1.]\n",
      " [1. 1. 1.]\n",
      " [1. 1. 1.]\n",
      " [1. 1. 1.]\n",
      " [1. 1. 1.]]\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Making a tensor filled with zeros. shape=[rows, columns]\n",
    "tensor = tf.zeros(shape=[3, 4], dtype=tf.int32)\n",
    "print(('Tensor full of zeros as int32, 3 rows and 4 columns:\\n{0}').format(\n",
    "    tensor.numpy()\n",
    "))\n",
    "\n",
    "# Making a tensor filled with zeros with data type of float32\n",
    "tensor = tf.ones(shape=[5, 3], dtype=tf.float32)\n",
    "print(('\\nTensor full of ones as float32, 5 rows and 3 columns:\\n{0}').format(\n",
    "    tensor.numpy()\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to reshape data with tf.reshape\n",
    "\n",
    "We might have generated some random noise or have a dataset of images in different sizes, which needs to be one-dimensional in order to fit into some filter or convolution.\n",
    "\n",
    "We could use `tf.reshape` to reshape the images in whichever way we want. All we do here is define a tensor, and then reshape it into 8 columns with 1 row, instead of 2 columns with 4 rows.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor BEFORE reshape:\n",
      "[[3 2]\n",
      " [5 2]\n",
      " [9 5]\n",
      " [1 3]]\n",
      "\n",
      "Tensor AFTER reshape:\n",
      "[[3 2 5 2 9 5 1 3]]\n"
     ]
    }
   ],
   "source": [
    "# Making a tensor for reshaping\n",
    "tensor = tf.constant([[3, 2],\n",
    "                      [5, 2],\n",
    "                      [9, 5],\n",
    "                      [1, 3]])\n",
    "\n",
    "# Reshaping the tensor into a shape of: shape = [rows, columns]\n",
    "reshaped_tensor = tf.reshape(tensor = tensor,\n",
    "                             shape = [1, 8])\n",
    "\n",
    "print(('Tensor BEFORE reshape:\\n{0}').format(\n",
    "    tensor.numpy()\n",
    "))\n",
    "print(('\\nTensor AFTER reshape:\\n{0}').format(\n",
    "    reshaped_tensor.numpy()\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to cast tensors to other data types with tf.cast\n",
    "\n",
    "Some functions in TensorFlow and Keras requires specific data types as inputs, and we can do that with tf.cast. If you mostly have integers, you will probably find yourself casting from integer values to float values.\n",
    "\n",
    "We can simply make a tensor with the datatype of `float32`. We can then cast this tensor to int, removing the comma and all decimals, while not rounding up or down.\n",
    "\n",
    "**NOTE** The output of this code piece will simply be stripping the commas from the original tensor to a new tensor without the commas – a successful conversion from `float` to `int`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor with floats:\n",
      "[[3.1 2.8]\n",
      " [5.2 2.3]\n",
      " [9.7 5.5]\n",
      " [1.1 3.4]]\n",
      "\n",
      "Tensor cast from float to int (just remove the decimal, no rounding):\n",
      "[[3 2]\n",
      " [5 2]\n",
      " [9 5]\n",
      " [1 3]]\n"
     ]
    }
   ],
   "source": [
    "# Making a tensor\n",
    "tensor = tf.constant([[3.1, 2.8],\n",
    "                      [5.2, 2.3],\n",
    "                      [9.7, 5.5],\n",
    "                      [1.1, 3.4]], \n",
    "                      dtype=tf.float32)\n",
    "\n",
    "tensor_as_int = tf.cast(tensor, tf.int32)\n",
    "\n",
    "print(('Tensor with floats:\\n{0}').format(\n",
    "    tensor.numpy()\n",
    "))\n",
    "print(('\\nTensor cast from float to int (just remove the decimal, no rounding):\\n{0}').format(\n",
    "    tensor_as_int.numpy()\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear Algebra Operations\n",
    "\n",
    "Many algorithms or research needs these operations in order to implement algorithms and trying new things, e.g. making smaller changes in activation functions or optimizers. \n",
    "\n",
    "- Transpose tensor with tf.transpose\n",
    "- Matrix Multiplication with tf.matmul\n",
    "- Element-wise multiplication with tf.multiply\n",
    "- Identity Matrix with tf.eye\n",
    "- Determinant with tf.linalg.det\n",
    "- Dot Product with tf.tensordot\n",
    "\n",
    "#### How to transpose a tensor with tf.transpose\n",
    "Suppose we want to do linear algebra operations, then the tf.transpose function comes in handy. This produces \n",
    "$A^T$\n",
    ", i.e. it produces the transposed matrix of A."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The transposed matrix A:\n",
      "[[3 1]\n",
      " [7 9]]\n"
     ]
    }
   ],
   "source": [
    "# Some Matrix A\n",
    "A = tf.constant([[3, 7],\n",
    "                 [1, 9]])\n",
    "\n",
    "A = tf.transpose(A)\n",
    "\n",
    "print(('The transposed matrix A:\\n{0}').format(A))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How to do matrix multiplication with tf.matmul\n",
    "\n",
    "Many algorithms requires matrix multiplication, and this is easy in TensorFlow with the tf.matmul function.\n",
    "\n",
    "All we do here is define two matrices (one is a vector) and use the tf.matmul function to do matrix multiplication."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matrix Multiplication of A and v results in a new Tensor:\n",
      "[[29]\n",
      " [23]]\n"
     ]
    }
   ],
   "source": [
    "# Some Matrix A\n",
    "A = tf.constant([[3, 7],\n",
    "                 [1, 9]])\n",
    "\n",
    "# Some vector v\n",
    "v = tf.constant([[5],\n",
    "                 [2]])\n",
    "\n",
    "# Matrix multiplication of A.v^T\n",
    "Av = tf.matmul(A, v)\n",
    "\n",
    "print(('Matrix Multiplication of A and v results in a new Tensor:\\n{0}').format(\n",
    "    Av\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How to do element-wise multiplication with `tf.multiply`\n",
    "\n",
    "Element-wise multiplication comes up in many instances, especially in optimizers. Reusing the tf.constants from before, such that we can compare the two, we simply use `tf.multiply` instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Element-wise multiplication of A and v results in a new Tensor:\n",
      "[[15 35]\n",
      " [ 2 18]]\n"
     ]
    }
   ],
   "source": [
    "# Element-wise multiplication\n",
    "Av = tf.multiply(A, v)\n",
    "\n",
    "print(('Element-wise multiplication of A and v results in a new Tensor:\\n{0}').format(\n",
    "    Av\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How to make an identity matrix with tf.eye\n",
    "In Linear Algebra, the identity matrix is simply a matrix with ones along the diagonal – and if you find the identity matrix of some matrix A, and multiply the identity matrix with A, the result will be the matrix A.\n",
    "\n",
    "We simply define a tensor A, get the rows and columns and make an identity matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Get rows and columns in tensor A:\n",
      "3 rows\n",
      "2 columns\n",
      "\n",
      "The identity matrix of A:\n",
      "[[1 0]\n",
      " [0 1]\n",
      " [0 0]]\n"
     ]
    }
   ],
   "source": [
    "# Some Matrix A\n",
    "A = tf.constant([[3, 7],\n",
    "                 [1, 9],\n",
    "                 [2, 5]])\n",
    "\n",
    "# Get number of dimensions\n",
    "rows, columns = A.shape\n",
    "print(('Get rows and columns in tensor A:\\n{0} rows\\n{1} columns').format(\n",
    "    rows, columns\n",
    "))\n",
    "\n",
    "# Making identity matrix\n",
    "A_identity = tf.eye(num_rows = rows,\n",
    "                    num_columns = columns,\n",
    "                    dtype = tf.int32)\n",
    "print(('\\nThe identity matrix of A:\\n{0}').format(\n",
    "    A_identity.numpy()\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How to find the determinant with tf.linalg.det\n",
    "The determinant can be used to solve linear equations or capturing how the area of how matrices changes.\n",
    "\n",
    "We make a matrix A, then cast it to float32, because the tf.linalg.det does not take integers as input. Then we just find the determinant of A."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The determinant of A:\n",
      "20.0\n"
     ]
    }
   ],
   "source": [
    "# Reusing Matrix A\n",
    "A = tf.constant([[3, 7],\n",
    "                 [1, 9]])\n",
    "\n",
    "# Determinant must be: half, float32, float64, complex64, complex128\n",
    "# Thus, we cast A to the data type float32\n",
    "A = tf.dtypes.cast(A, tf.float32)\n",
    "\n",
    "# Finding the determinant of A\n",
    "det_A = tf.linalg.det(A)\n",
    "\n",
    "print(('The determinant of A:\\n{0}').format(\n",
    "    det_A\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How to find the dot product with tf.tensordot\n",
    "Dotting one tensor onto another is perhaps one of the most common linear algebra operations. Hence, we should at least know how to find the dot product of two tenors in TensorFlow.\n",
    "\n",
    "We just need to instantiate two constants, and then we can dot them together – note that in this instance, tf.tensordot is the same as tf.matmul, but there are differences outside the scope of this article."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining a 3x3 matrix\n",
    "A = tf.constant([[32, 83, 5],\n",
    "                 [17, 23, 10],\n",
    "                 [75, 39, 52]])\n",
    "\n",
    "# Defining another 3x3 matrix\n",
    "B = tf.constant([[28, 57, 20],\n",
    "                 [91, 10, 95],\n",
    "                 [37, 13, 45]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dot product of A.B^T gives a new Tensor:\n",
      "[[8634 2719 8750]\n",
      " [2939 1329 2975]\n",
      " [7573 5341 7545]]\n"
     ]
    }
   ],
   "source": [
    "# Getting dot product\n",
    "dot_AB = tf.tensordot(a=A, b=B, axes=1).numpy()\n",
    "print(('Dot product of A.B^T gives a new Tensor:\\n{0}'). format(dot_AB))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Matrix Multiplication of A.B^T results in a new Tensor:\n",
      "[[8634 2719 8750]\n",
      " [2939 1329 2975]\n",
      " [7573 5341 7545]]\n"
     ]
    }
   ],
   "source": [
    "# Which is the same as matrix multiplication in this instance (axes=1)\n",
    "# Matrix multiplication of A and B\n",
    "AB = tf.matmul(A, B)\n",
    "\n",
    "print(('\\nMatrix Multiplication of A.B^T results in a new Tensor:\\n{0}').format(\n",
    "    AB\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculating Gradients\n",
    "\n",
    "Let's make an example of the newer GELU activation function, used in OpenAI's GPT-2 and Google's BERT.\n",
    "\n",
    "The GELU function:\n",
    "\n",
    "$$\n",
    "\\operatorname{GELU}(x)=0.5 x\\left(1+\\tanh \\left(\\sqrt{2 / \\pi}\\left(x+0.044715 x^{3}\\right)\\right)\\right)\n",
    "$$\n",
    "\n",
    "GELU differentiated:\n",
    "\n",
    "$\n",
    "\\begin{aligned}\n",
    "\\operatorname{GELU}^{\\prime}(x) &=0.5 \\tanh \\left(0.0356774 x^{3}+0.79785*0.5\\right) \\\\\n",
    "+\\left(0.0535161* 0.5^{3}+0.398942*0.5\\right) \\operatorname{sech}^{2}\\left(0.0356774*0.5^{3}+0.797885*0.5\\right) &+0.5 = 0.867370\n",
    "\\end{aligned}\n",
    "$\n",
    "\n",
    "\n",
    "![Image of GELU](https://mlfromscratch.com/content/images/2019/12/image-4.png)\n",
    "\n",
    "#### Let's just code this into an example in TensorFlow.\n",
    "\n",
    "First, define the activation function; we chose the GELU activation function `gelu()`. Then we define a `get_gradient()` function which uses the Gradient Tape from TensorFlow.\n",
    "\n",
    "The **Gradient Tape** is the important part, since it automatically differentiates and records the gradient of any operation indented under `tf.GradientTape() as gt` . After execution, we use the gradient tape with the gradient function gt.gradient() to retrieve the recorded gradient for the target y from the source x."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8673699498176575 is the gradient of GELU with x=0.5\n"
     ]
    }
   ],
   "source": [
    "import math\n",
    "\n",
    "def gelu(x):\n",
    "    return 0.5*x*(1+tf.tanh(tf.sqrt(2/math.pi)*(x+0.044715*tf.pow(x, 3))))\n",
    "\n",
    "def get_gradient(x, activation_function):\n",
    "    with tf.GradientTape() as gt:\n",
    "        y = activation_function(x)\n",
    "\n",
    "    gradient = gt.gradient(y, x).numpy()\n",
    "\n",
    "    return gradient\n",
    "\n",
    "x = tf.Variable(0.5)\n",
    "gradient = get_gradient(x, gelu)\n",
    "\n",
    "print('{0} is the gradient of GELU with x={1}'.format(\n",
    "    gradient, x.numpy()\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Functions In TensorFlow 2.0\n",
    "\n",
    "TensorFlow Functions with `@tf.function` offers a significant speedup, because TensorFlow uses AutoGraph to convert functions to graphs, which in turn runs faster.\n",
    "\n",
    "The annotation takes the normal Python syntax and converts it into a graph – and it has minimum side effects, which means we should always use it, especially when training and testing neural network models.\n",
    "\n",
    "All that is done here is making an image and running it through `conv_layer` and `conv_fn`, then finding the difference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Without tf.function:  0.40485883499999886\n",
      "With tf.function:  0.37121672400000705\n",
      "The difference:  0.033642110999991814\n",
      "\n",
      "Just imagine when we have to do millions/billions of these calculations, then the difference will be HUGE!\n",
      "Difference times a billion:  33642110.99999181\n"
     ]
    }
   ],
   "source": [
    "import timeit\n",
    "conv_layer = tf.keras.layers.Conv2D(100, 3)\n",
    "\n",
    "@tf.function\n",
    "def conv_fn(image):\n",
    "  return conv_layer(image)\n",
    "\n",
    "image = tf.zeros([1, 200, 200, 100])\n",
    "# warm up\n",
    "conv_layer(image); conv_fn(image)\n",
    "\n",
    "no_tf_fn = timeit.timeit(lambda: conv_layer(image), number=10)\n",
    "with_tf_fn = timeit.timeit(lambda: conv_fn(image), number=10)\n",
    "difference = no_tf_fn - with_tf_fn\n",
    "\n",
    "print(\"Without tf.function: \", no_tf_fn)\n",
    "print(\"With tf.function: \", with_tf_fn)\n",
    "print(\"The difference: \", difference)\n",
    "print(\"\\nJust imagine when we have to do millions/billions of these calculations,\" \\\n",
    "      \" then the difference will be HUGE!\")\n",
    "print(\"Difference times a billion: \", difference*1000000000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom Train and Test Functions In TensorFlow 2.0\n",
    "\n",
    "For this part, we are going to be following a heavily modified approach of the tutorial from tensorflow's documentation.\n",
    "\n",
    "For the first part, we just have some imports that we need for later. We also specify that the backend should by default run float64 in layers.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "#STEP1 --> Import stuff\n",
    "\n",
    "from __future__ import absolute_import, division, print_function, unicode_literals\n",
    "\n",
    "from tensorflow.keras.layers import Dense, Flatten, Conv2D\n",
    "from tensorflow.keras import Model\n",
    "\n",
    "tf.keras.backend.set_floatx('float64')\n",
    "\n",
    "mnist = tf.keras.datasets.mnist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "#STEP 2 --> Load Data & Remove color channels\n",
    "# Load Data & Remove color channels\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "x_train, x_test = x_train / 255.0, x_test / 255.0\n",
    "\n",
    "# Add a channels dimension\n",
    "x_train = x_train[..., tf.newaxis]\n",
    "x_test = x_test[..., tf.newaxis]\n",
    "\n",
    "train_ds = tf.data.Dataset.from_tensor_slices(\n",
    "    (x_train, y_train)).shuffle(10000).batch(32)\n",
    "\n",
    "test_ds = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we make a class, which starts here and each function will be described in it's separate little code piece.\n",
    "\n",
    "If you don't know what an `__init__()` function does, then all you need to know is that a **constructor** – a constructor runs this the code in it's function `__init__` every time you instantiate (explained later) a new object of that class. The first step in TensorFlow is using the `super()` function, to run the superclass of the current subclass. All other code is a standard approach, we just define some variables and layers, like convolutions and dense layers. When we use the self., we assign a variable to the instance of the class, such that we can do `self.conv1` in other methods, and we can do `MyModel.conv1` outside the class, to access that specific variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "class MyModel(Model):\n",
    "    def __init__(self,\n",
    "                 loss_object,\n",
    "                 optimizer,\n",
    "                 train_loss,\n",
    "                 train_metric,\n",
    "                 test_loss,\n",
    "                 test_metric):\n",
    "        '''\n",
    "            Setting all the variables for our model.\n",
    "        '''\n",
    "        super(MyModel, self).__init__()\n",
    "        self.conv1 = Conv2D(32, 3, activation='relu')\n",
    "        self.flatten = Flatten()\n",
    "        self.d1 = Dense(128, activation='relu')\n",
    "        self.d2 = Dense(10, activation='softmax')\n",
    "\n",
    "        self.loss_object = loss_object\n",
    "        self.optimizer = optimizer\n",
    "        self.train_loss = train_loss\n",
    "        self.train_metric = train_metric\n",
    "        self.test_loss = test_loss\n",
    "        self.test_metric = test_metric\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next function is defining the architecture for our neural network, hence why it's called `nn_model()`. We just run through the model here when it's called with some input x. One smaller exercise, if you are just getting started out with Python/TensorFlow would be to remove the function nn_model, and provide it as an input when instantiating the class. Remember to replace references with the new name you give it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "def nn_model(self, x):\n",
    "        '''\n",
    "            Defining the architecture of our model. This is where we run \n",
    "            through our whole dataset and return it, when training and \n",
    "            testing.\n",
    "        '''\n",
    "        x = self.conv1(x)\n",
    "        x = self.flatten(x)\n",
    "        x = self.d1(x)\n",
    "        return self.d2(x)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's watch really close, lots of things are happening in the next function. First of all, we annotated the function with @tf.function for as much of a speedup as possible.\n",
    "\n",
    "As explained earlier, the `tf.GradientTape()` records gradients onto a variable tape, which we can access afterwards. The training goes like this:\n",
    "\n",
    "1. Make predictions and call the object holding the loss function with our data and predictions. While this is happening, gradients were automatically recorded.\n",
    "2. Get the gradients from the gradient tape and apply them using the update rule from the optimizer picked (we will look at inputting these functions and variables later).\n",
    "3. As for the zip\n",
    "\n",
    "\"I don't know Tensorflow, but presumably `optimizer.compute_gradients(loss)` yields (gradient, value) tuples.\n",
    "\n",
    "`gradients, v = zip(*optimizer.compute_gradients(loss))`\n",
    "performs a **transposition**, creating a **list of gradients and a list of values**.\n",
    "\n",
    "`gradients, _ = tf.clip_by_global_norm(gradients, 1.25)`\n",
    "then clips the gradients, and\n",
    "\n",
    "`optimizer = optimizer.apply_gradients(zip(gradients, v), global_step=global_step)`\n",
    "re-zips the gradient and value lists back into an **iterable of (gradient, value) tuples** which is then passed to the `optimizer.apply_gradients` method.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "@tf.function\n",
    "def train_step(self, images, labels):\n",
    "        '''\n",
    "            This is a TensorFlow function, run once for each epoch for the\n",
    "            whole input. We move forward first, then calculate gradients \n",
    "            with Gradient Tape to move backwards.\n",
    "        '''\n",
    "        with tf.GradientTape() as tape:\n",
    "            predictions = self.nn_model(images)\n",
    "            loss = self.loss_object(labels, predictions)\n",
    "        gradients = tape.gradient(loss, self.trainable_variables)\n",
    "        optimizer.apply_gradients(zip(\n",
    "                                  gradients, self.trainable_variables))\n",
    "\n",
    "        self.train_loss(loss)\n",
    "        self.train_metric(labels, predictions)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This next function is just a test step, used to test the last training step. This function is almost identical to the train_step() function, except for there are no gradients and updates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "@tf.function\n",
    "def test_step(self, images, labels):\n",
    "        '''\n",
    "            This is a TensorFlow function, run once for each epoch for the\n",
    "            whole input.\n",
    "        '''\n",
    "        predictions = self.nn_model(images)\n",
    "        t_loss = self.loss_object(labels, predictions)\n",
    "\n",
    "        self.test_loss(t_loss)\n",
    "        self.test_metric(labels, predictions)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next function ties the whole class together into one function, with three for loops. Later on, we define how many epochs (iterations) we want the neural networks to train and test for – and then for each iteration, we run through each observation.\n",
    "\n",
    "Afterwards, we can see how well we optimized our loss function and metric. We just keep running this from \n",
    "`0` to `n` epochs. This concludes the class `MyModel`. Have a close look at the three for loops, as that is where all the action is happening."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "def fit(self, train, test, epochs):\n",
    "        '''\n",
    "            This fit function runs training and testing.\n",
    "        '''\n",
    "        for epoch in range(epochs):\n",
    "            for images, labels in train:\n",
    "                self.train_step(images, labels)\n",
    "\n",
    "            for test_images, test_labels in test:\n",
    "                self.test_step(test_images, test_labels)\n",
    "\n",
    "            template = 'Epoch {}, Loss: {}, Accuracy: {}, Test Loss: {}, Test Accuracy: {}'\n",
    "            print(template.format(epoch+1,\n",
    "                                  self.train_loss.result(),\n",
    "                                  self.train_metric.result()*100,\n",
    "                                  self.test_loss.result(),\n",
    "                                  self.test_metric.result()*100))\n",
    "\n",
    "            # Reset the metrics for the next epoch\n",
    "            self.train_loss.reset_states()\n",
    "            self.train_metric.reset_states()\n",
    "            self.test_loss.reset_states()\n",
    "            self.test_metric.reset_states()\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 3 --> Build Model Piecing this all together\n",
    "class MyModel(Model):\n",
    "    def __init__(self,\n",
    "                 loss_object,\n",
    "                 optimizer,\n",
    "                 train_loss,\n",
    "                 train_metric,\n",
    "                 test_loss,\n",
    "                 test_metric):\n",
    "        '''\n",
    "            Setting all the variables for our model.\n",
    "        '''\n",
    "        super(MyModel, self).__init__()\n",
    "        self.conv1 = Conv2D(32, 3, activation='relu')\n",
    "        self.flatten = Flatten()\n",
    "        self.d1 = Dense(128, activation='relu')\n",
    "        self.d2 = Dense(10, activation='softmax')\n",
    "\n",
    "        self.loss_object = loss_object\n",
    "        self.optimizer = optimizer\n",
    "        self.train_loss = train_loss\n",
    "        self.train_metric = train_metric\n",
    "        self.test_loss = test_loss\n",
    "        self.test_metric = test_metric\n",
    "\n",
    "    def nn_model(self, x):\n",
    "        '''\n",
    "            Defining the architecture of our model. This is where we run \n",
    "            through our whole dataset and return it, when training and testing.\n",
    "        '''\n",
    "        x = self.conv1(x)\n",
    "        x = self.flatten(x)\n",
    "        x = self.d1(x)\n",
    "        return self.d2(x)\n",
    "    \n",
    "    @tf.function\n",
    "    def train_step(self, images, labels):\n",
    "        '''\n",
    "            This is a TensorFlow function, run once for each epoch for the\n",
    "            whole input. We move forward first, then calculate gradients with\n",
    "            Gradient Tape to move backwards.\n",
    "        '''\n",
    "        with tf.GradientTape() as tape:\n",
    "            predictions = self.nn_model(images)\n",
    "            loss = self.loss_object(labels, predictions)\n",
    "        gradients = tape.gradient(loss, self.trainable_variables)\n",
    "        optimizer.apply_gradients(zip(gradients, self.trainable_variables))\n",
    "\n",
    "        self.train_loss(loss)\n",
    "        self.train_metric(labels, predictions)\n",
    "\n",
    "    @tf.function\n",
    "    def test_step(self, images, labels):\n",
    "        '''\n",
    "            This is a TensorFlow function, run once for each epoch for the\n",
    "            whole input.\n",
    "        '''\n",
    "        predictions = self.nn_model(images)\n",
    "        t_loss = self.loss_object(labels, predictions)\n",
    "\n",
    "        self.test_loss(t_loss)\n",
    "        self.test_metric(labels, predictions)\n",
    "    \n",
    "    def fit(self, train, test, epochs):\n",
    "        '''\n",
    "            This fit function runs training and testing.\n",
    "        '''\n",
    "        for epoch in range(epochs):\n",
    "            for images, labels in train:\n",
    "                self.train_step(images, labels)\n",
    "\n",
    "            for test_images, test_labels in test:\n",
    "                self.test_step(test_images, test_labels)\n",
    "\n",
    "            template = 'Epoch {}, Loss: {}, Accuracy: {}, Test Loss: {}, Test Accuracy: {}'\n",
    "            print(template.format(epoch+1,\n",
    "                                  self.train_loss.result(),\n",
    "                                  self.train_metric.result()*100,\n",
    "                                  self.test_loss.result(),\n",
    "                                  self.test_metric.result()*100))\n",
    "\n",
    "            # Reset the metrics for the next epoch\n",
    "            self.train_loss.reset_states()\n",
    "            self.train_metric.reset_states()\n",
    "            self.test_loss.reset_states()\n",
    "            self.test_metric.reset_states()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 4 --> Loss, Optimizers \n",
    "# For the next snippet of code, we simply define all the\n",
    "# variables and functions we need for a neural network to \n",
    "# run – a loss function, optimizer and metric.\n",
    "\n",
    "# Make a loss object\n",
    "loss_object = tf.keras.losses.SparseCategoricalCrossentropy()\n",
    "\n",
    "# Select the optimizer\n",
    "optimizer = tf.keras.optimizers.Adam()\n",
    "\n",
    "# Specify metrics for training\n",
    "train_loss = tf.keras.metrics.Mean(name='train_loss')\n",
    "train_metric = tf.keras.metrics.SparseCategoricalAccuracy(name='train_accuracy')\n",
    "\n",
    "# Specify metrics for testing\n",
    "test_loss = tf.keras.metrics.Mean(name='test_loss')\n",
    "test_metric = tf.keras.metrics.SparseCategoricalAccuracy(name='test_accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We take the loss functions, optimizer and metrics, and we input that into MyModel by instantiating the class with these variables. So when we call `MyModel()` with all these parameters, we actually run the `__init__` function in the `MyModel` class.\n",
    "\n",
    "As mentioned earlier, we can call functions and variables from the instance of a class, so here we quite simply call the fit function with our training and testing dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1, Loss: 0.13795717211390535, Accuracy: 95.82000000000001, Test Loss: 0.06255114574318962, Test Accuracy: 98.0\n",
      "Epoch 2, Loss: 0.04473557519063664, Accuracy: 98.61333333333333, Test Loss: 0.05246945184844895, Test Accuracy: 98.18\n",
      "Epoch 3, Loss: 0.022526336245403702, Accuracy: 99.245, Test Loss: 0.059170655907106996, Test Accuracy: 98.1\n",
      "Epoch 4, Loss: 0.01435684646053851, Accuracy: 99.555, Test Loss: 0.055213415196283255, Test Accuracy: 98.34\n",
      "Epoch 5, Loss: 0.01020043003212098, Accuracy: 99.65833333333333, Test Loss: 0.0601451760462268, Test Accuracy: 98.42\n",
      "Epoch 6, Loss: 0.007858400384669737, Accuracy: 99.73166666666667, Test Loss: 0.07318530304548679, Test Accuracy: 98.25\n",
      "Epoch 7, Loss: 0.005572388626881517, Accuracy: 99.82166666666666, Test Loss: 0.06766302098721277, Test Accuracy: 98.4\n",
      "Epoch 8, Loss: 0.004817213012027575, Accuracy: 99.82166666666666, Test Loss: 0.08149071166639954, Test Accuracy: 98.28\n",
      "Epoch 9, Loss: 0.003920269912300882, Accuracy: 99.86500000000001, Test Loss: 0.08439076490274913, Test Accuracy: 98.32\n",
      "Epoch 10, Loss: 0.003568455249099412, Accuracy: 99.89666666666666, Test Loss: 0.08385327842149685, Test Accuracy: 98.34\n",
      "Epoch 11, Loss: 0.004465888586936313, Accuracy: 99.85166666666667, Test Loss: 0.08196696018479167, Test Accuracy: 98.33\n",
      "Epoch 12, Loss: 0.00275107173654903, Accuracy: 99.89833333333334, Test Loss: 0.08720286394619745, Test Accuracy: 98.3\n",
      "Epoch 13, Loss: 0.0029471971701518452, Accuracy: 99.9, Test Loss: 0.07805522627763155, Test Accuracy: 98.49\n",
      "Epoch 14, Loss: 0.002791931772175399, Accuracy: 99.91333333333333, Test Loss: 0.08921483688237668, Test Accuracy: 98.13\n",
      "Epoch 15, Loss: 0.0024399050318205874, Accuracy: 99.925, Test Loss: 0.0966548980436959, Test Accuracy: 98.44000000000001\n",
      "Epoch 16, Loss: 0.002400517416392249, Accuracy: 99.92, Test Loss: 0.10152959187355697, Test Accuracy: 98.42\n",
      "Epoch 17, Loss: 0.001995964271829157, Accuracy: 99.93166666666666, Test Loss: 0.09195649219075348, Test Accuracy: 98.58\n",
      "Epoch 18, Loss: 0.002583709886227739, Accuracy: 99.92999999999999, Test Loss: 0.1075754314355452, Test Accuracy: 98.4\n",
      "Epoch 19, Loss: 0.0016927552294467563, Accuracy: 99.94166666666666, Test Loss: 0.10530795230149022, Test Accuracy: 98.39\n",
      "Epoch 20, Loss: 0.0005705737490631062, Accuracy: 99.97666666666667, Test Loss: 0.10486208517044106, Test Accuracy: 98.57000000000001\n",
      "Epoch 21, Loss: 0.0026385881917003143, Accuracy: 99.91666666666667, Test Loss: 0.09607810602462885, Test Accuracy: 98.49\n",
      "Epoch 22, Loss: 0.0011588127385430192, Accuracy: 99.965, Test Loss: 0.10563130390731879, Test Accuracy: 98.38\n",
      "Epoch 23, Loss: 0.0009087644932885645, Accuracy: 99.96666666666667, Test Loss: 0.10402541684756438, Test Accuracy: 98.49\n",
      "Epoch 24, Loss: 0.0015104768340796018, Accuracy: 99.95833333333334, Test Loss: 0.14016618319184587, Test Accuracy: 98.26\n",
      "Epoch 25, Loss: 0.0014181080629266807, Accuracy: 99.96333333333334, Test Loss: 0.13292021209523602, Test Accuracy: 98.37\n",
      "Epoch 26, Loss: 0.002159601998991138, Accuracy: 99.93833333333333, Test Loss: 0.15958019538093723, Test Accuracy: 98.18\n",
      "Epoch 27, Loss: 0.0006434448424455876, Accuracy: 99.97833333333334, Test Loss: 0.1359496810136438, Test Accuracy: 98.52\n",
      "Epoch 28, Loss: 0.0013669618745628168, Accuracy: 99.96000000000001, Test Loss: 0.14913920077520584, Test Accuracy: 98.35000000000001\n",
      "Epoch 29, Loss: 0.001675322440243567, Accuracy: 99.94833333333332, Test Loss: 0.16362160408417123, Test Accuracy: 98.05\n",
      "Epoch 30, Loss: 0.0008937887896611205, Accuracy: 99.97666666666667, Test Loss: 0.1479875487857637, Test Accuracy: 98.42999999999999\n"
     ]
    }
   ],
   "source": [
    "# Step 5 --> Create an instance of the model and run\n",
    "model = MyModel(loss_object = loss_object,\n",
    "                optimizer = optimizer,\n",
    "                train_loss = train_loss,\n",
    "                train_metric = train_metric,\n",
    "                test_loss = test_loss,\n",
    "                test_metric = test_metric)\n",
    "\n",
    "EPOCHS = 30\n",
    "\n",
    "model.fit(train = train_ds,\n",
    "          test = test_ds,\n",
    "          epochs = EPOCHS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}