creating_and_manipulating_tensors.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "creating_and_manipulating_tensors.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [
        "JndnmDMp66FL",
        "EYzU56M4MG_x",
        "Kt7aojXkR_qS"
      ],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "[View in Colaboratory](https://colab.research.google.com/github/ArunkumarRamanan/Exercises-Machine-Learning-Crash-Course-Google-Developers/blob/master/creating_and_manipulating_tensors.ipynb)"
      ]
    },
    {
      "metadata": {
        "id": "JndnmDMp66FL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Copyright 2017 Google LLC."
      ]
    },
    {
      "metadata": {
        "id": "hMqWDc_m6rUC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "P0bQXjp499sl",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Creating and Manipulating Tensors"
      ]
    },
    {
      "metadata": {
        "id": "I3BCiWJiCGsv",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Learning Objectives:**\n",
        "  * Initialize and assign TensorFlow `Variable`s\n",
        "  * Create and manipulate tensors\n",
        "  * Refresh your memory about addition and multiplication in linear algebra (consult an introduction to matrix [addition](https://en.wikipedia.org/wiki/Matrix_addition) and [multiplication](https://en.wikipedia.org/wiki/Matrix_multiplication) if these topics are new to you)\n",
        "  * Familiarize yourself with basic TensorFlow math and array operations"
      ]
    },
    {
      "metadata": {
        "id": "85evKRsOIC5a",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import tensorflow as tf"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PT1sorfH-DdQ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Vector Addition\n",
        "\n",
        "You can perform many typical mathematical operations on tensors ([TF API](https://www.tensorflow.org/api_guides/python/math_ops)). The following code\n",
        "creates and manipulates two vectors (1-D tensors), each having exactly six elements:"
      ]
    },
    {
      "metadata": {
        "id": "ng37e6ur-GZo",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "with tf.Graph().as_default():\n",
        "  # Create a six-element vector (1-D tensor).\n",
        "  primes = tf.constant([2, 3, 5, 7, 11, 13], dtype=tf.int32)\n",
        "\n",
        "  # Create another six-element vector. Each element in the vector will be\n",
        "  # initialized to 1. The first argument is the shape of the tensor (more\n",
        "  # on shapes below).\n",
        "  ones = tf.ones([6], dtype=tf.int32)\n",
        "\n",
        "  # Add the two vectors. The resulting tensor is a six-element vector.\n",
        "  just_beyond_primes = tf.add(primes, ones)\n",
        "\n",
        "  # Create a session to run the default graph.\n",
        "  with tf.Session() as sess:\n",
        "    print just_beyond_primes.eval()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "fVvaXzzMGZid",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Tensor Shapes\n",
        "\n",
        "Shapes are used to characterize the size and number of dimensions of a tensor. The shape of a tensor is expressed as `list`, with the `i`th element representing the size along dimension `i`. The length of the list then indicates the rank of the tensor (i.e., the number of dimensions).\n",
        "\n",
        "For more information, see the [TensorFlow documentation](https://www.tensorflow.org/programmers_guide/tensors#shape).\n",
        "\n",
        "A few basic examples:"
      ]
    },
    {
      "metadata": {
        "id": "PWzvJnIAH_cF",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "with tf.Graph().as_default():\n",
        "  # A scalar (0-D tensor).\n",
        "  scalar = tf.zeros([])\n",
        "\n",
        "  # A vector with 3 elements.\n",
        "  vector = tf.zeros([3])\n",
        "\n",
        "  # A matrix with 2 rows and 3 columns.\n",
        "  matrix = tf.zeros([2, 3])\n",
        "\n",
        "  with tf.Session() as sess:\n",
        "    print 'scalar has shape', scalar.get_shape(), 'and value:\\n', scalar.eval()\n",
        "    print 'vector has shape', vector.get_shape(), 'and value:\\n', vector.eval()\n",
        "    print 'matrix has shape', matrix.get_shape(), 'and value:\\n', matrix.eval()\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "musamrLavR5S",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Broadcasting\n",
        "\n",
        "In mathematics, you can only perform element-wise operations (e.g. *add* and *equals*) on tensors of the same shape. In TensorFlow, however, you may perform operations on tensors that would traditionally have been incompatible. TensorFlow supports **broadcasting** (a concept borrowed from numpy), where the smaller array in an element-wise operation is enlarged to have the same shape as the larger array. For example, via broadcasting:\n",
        "\n",
        "* If an operand requires a size `[6]` tensor, a size `[1]` or a size `[]` tensor can serve as an operand.\n",
        "* If an operation requires a size `[4, 6]` tensor, any of the following sizes can serve as an operand:\n",
        "  * `[1, 6]`\n",
        "  * `[6]`\n",
        "  * `[]`\n",
        "* If an operation requires a size `[3, 5, 6]` tensor, any of the following sizes can serve as an operand:\n",
        "\n",
        "  * `[1, 5, 6]`\n",
        "  * `[3, 1, 6]`\n",
        "  * `[3, 5, 1]`\n",
        "  * `[1, 1, 1]`\n",
        "  * `[5, 6]`\n",
        "  * `[1, 6]`\n",
        "  * `[6]`\n",
        "  * `[1]`\n",
        "  * `[]`\n",
        "  \n",
        "**NOTE:** When a tensor is broadcast, its entries are conceptually **copied**. (They are not actually copied for performance reasons. Broadcasting was invented as a performance optimization.)\n",
        "\n",
        "The full broadcasting ruleset is well described in the easy-to-read [numpy broadcasting documentation](http://docs.scipy.org/doc/numpy-1.10.1/user/basics.broadcasting.html).\n",
        "\n",
        "The following code performs the same tensor addition as before, but using broadcasting:"
      ]
    },
    {
      "metadata": {
        "id": "7lys_BeLy2SD",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "with tf.Graph().as_default():\n",
        "  # Create a six-element vector (1-D tensor).\n",
        "  primes = tf.constant([2, 3, 5, 7, 11, 13], dtype=tf.int32)\n",
        "\n",
        "  # Create a constant scalar with value 1.\n",
        "  ones = tf.constant(1, dtype=tf.int32)\n",
        "\n",
        "  # Add the two tensors. The resulting tensor is a six-element vector.\n",
        "  just_beyond_primes = tf.add(primes, ones)\n",
        "\n",
        "  with tf.Session() as sess:\n",
        "    print just_beyond_primes.eval()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "it0P-AV0-Jb4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Matrix Multiplication\n",
        "\n",
        "In linear algebra, when multiplying two matrices, the number of *columns* of the first matrix must\n",
        "equal the number of *rows* in the second matrix.\n",
        "\n",
        "- It is **_valid_** to multiply a `3x4` matrix by a `4x2` matrix. This will result in a `3x2` matrix.\n",
        "- It is **_invalid_** to multiply a `4x2` matrix by a `3x4` matrix."
      ]
    },
    {
      "metadata": {
        "id": "OVR8QPif-MeS",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "with tf.Graph().as_default():\n",
        "  # Create a matrix (2-d tensor) with 3 rows and 4 columns.\n",
        "  x = tf.constant([[5, 2, 4, 3], [5, 1, 6, -2], [-1, 3, -1, -2]],\n",
        "                  dtype=tf.int32)\n",
        "\n",
        "  # Create a matrix with 4 rows and 2 columns.\n",
        "  y = tf.constant([[2, 2], [3, 5], [4, 5], [1, 6]], dtype=tf.int32)\n",
        "\n",
        "  # Multiply `x` by `y`. \n",
        "  # The resulting matrix will have 3 rows and 2 columns.\n",
        "  matrix_multiply_result = tf.matmul(x, y)\n",
        "\n",
        "  with tf.Session() as sess:\n",
        "    print matrix_multiply_result.eval()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "fziRnmuy-O9x",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Tensor Reshaping\n",
        "\n",
        "With tensor addition and matrix multiplication each imposing constraints\n",
        "on operands, TensorFlow programmers must frequently reshape tensors. \n",
        "\n",
        "You can use the `tf.reshape` method to reshape a tensor. \n",
        "For example, you can reshape a 8x2 tensor into a 2x8 tensor or a 4x4 tensor:"
      ]
    },
    {
      "metadata": {
        "id": "L05ob6a_G77m",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "with tf.Graph().as_default():\n",
        "  # Create an 8x2 matrix (2-D tensor).\n",
        "  matrix = tf.constant([[1,2], [3,4], [5,6], [7,8],\n",
        "                        [9,10], [11,12], [13, 14], [15,16]], dtype=tf.int32)\n",
        "\n",
        "  # Reshape the 8x2 matrix into a 2x8 matrix.\n",
        "  reshaped_2x8_matrix = tf.reshape(matrix, [2,8])\n",
        "  \n",
        "  # Reshape the 8x2 matrix into a 4x4 matrix\n",
        "  reshaped_4x4_matrix = tf.reshape(matrix, [4,4])\n",
        "\n",
        "  with tf.Session() as sess:\n",
        "    print \"Original matrix (8x2):\"\n",
        "    print matrix.eval()\n",
        "    print \"Reshaped matrix (2x8):\"\n",
        "    print reshaped_2x8_matrix.eval()\n",
        "    print \"Reshaped matrix (4x4):\"\n",
        "    print reshaped_4x4_matrix.eval()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "b6cFa92YGyU5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "You can also use `tf.reshape` to change the number of dimensions (the \"rank\") of the tensor.\n",
        "For example, you could reshape that 8x2 tensor into a 3-D 2x2x4 tensor or a 1-D 16-element tensor."
      ]
    },
    {
      "metadata": {
        "id": "3MpcwWj9-Sqp",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "with tf.Graph().as_default():\n",
        "  # Create an 8x2 matrix (2-D tensor).\n",
        "  matrix = tf.constant([[1,2], [3,4], [5,6], [7,8],\n",
        "                        [9,10], [11,12], [13, 14], [15,16]], dtype=tf.int32)\n",
        "\n",
        "  # Reshape the 8x2 matrix into a 3-D 2x2x4 tensor.\n",
        "  reshaped_2x2x4_tensor = tf.reshape(matrix, [2,2,4])\n",
        "  \n",
        "  # Reshape the 8x2 matrix into a 1-D 16-element tensor.\n",
        "  one_dimensional_vector = tf.reshape(matrix, [16])\n",
        "\n",
        "  with tf.Session() as sess:\n",
        "    print \"Original matrix (8x2):\"\n",
        "    print matrix.eval()\n",
        "    print \"Reshaped 3-D tensor (2x2x4):\"\n",
        "    print reshaped_2x2x4_tensor.eval()\n",
        "    print \"1-D vector:\"\n",
        "    print one_dimensional_vector.eval()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "CrpowaWo-VLq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Exercise #1: Reshape two tensors in order to multiply them.\n",
        "\n",
        "The following two vectors are incompatible for matrix multiplication:\n",
        "\n",
        "  *  `a = tf.constant([5, 3, 2, 7, 1, 4])`\n",
        "  *  `b = tf.constant([4, 6, 3])`\n",
        "\n",
        "Reshape these vectors into compatible operands for matrix multiplication.\n",
        "Then, invoke a matrix multiplication operation on the reshaped tensors."
      ]
    },
    {
      "metadata": {
        "id": "p6idvaeK-Zxq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "  # Write your code for Task 1 here."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "EYzU56M4MG_x",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Solution\n",
        "\n",
        "Click below for a solution."
      ]
    },
    {
      "metadata": {
        "id": "8Sef4d0SMMtk",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "with tf.Graph().as_default(), tf.Session() as sess:\n",
        "  # Task: Reshape two tensors in order to multiply them\n",
        "  \n",
        "  # Here are the original operands, which are incompatible\n",
        "  # for matrix multiplication:\n",
        "  a = tf.constant([5, 3, 2, 7, 1, 4])\n",
        "  b = tf.constant([4, 6, 3])\n",
        "  # We need to reshape at least one of these operands so that\n",
        "  # the number of columns in the first operand equals the number\n",
        "  # of rows in the second operand.\n",
        "\n",
        "  # Reshape vector \"a\" into a 2-D 2x3 matrix:\n",
        "  reshaped_a = tf.reshape(a, [2,3])\n",
        "\n",
        "  # Reshape vector \"b\" into a 2-D 3x1 matrix:\n",
        "  reshaped_b = tf.reshape(b, [3,1])\n",
        "\n",
        "  # The number of columns in the first matrix now equals\n",
        "  # the number of rows in the second matrix. Therefore, you\n",
        "  # can matrix mutiply the two operands.\n",
        "  c = tf.matmul(reshaped_a, reshaped_b)\n",
        "  print(c.eval())\n",
        "\n",
        "  # An alternate approach: [6,1] x [1, 3] -> [6,3]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "x1JYo7iE2oKk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Variables, Initialization and Assignment\n",
        "\n",
        "So far, all the operations we performed were on static values (`tf.constant`); calling `eval()` always returned the same result. TensorFlow allows you to define `Variable` objects, whose values can be changed. \n",
        "\n",
        "When creating a variable, you can set an initial value explicitly, or you can use an initializer (like a distribution):"
      ]
    },
    {
      "metadata": {
        "id": "6opLnjfD3PdL",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "g = tf.Graph()\n",
        "with g.as_default():\n",
        "  # Create a variable with the initial value 3.\n",
        "  v = tf.Variable([3])\n",
        "\n",
        "  # Create a variable of shape [1], with a random initial value,\n",
        "  # sampled from a normal distribution with mean 1 and standard deviation 0.35.\n",
        "  w = tf.Variable(tf.random_normal([1], mean=1.0, stddev=0.35))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qDYRXHTA4PTB",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "One peculiarity of TensorFlow is that **variable initialization is not automatic**. For example, the following block will cause an error:"
      ]
    },
    {
      "metadata": {
        "id": "d0OX1YRY5PTP",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "with g.as_default():\n",
        "  with tf.Session() as sess:\n",
        "    try:\n",
        "      v.eval()\n",
        "    except tf.errors.FailedPreconditionError as e:\n",
        "      print \"Caught expected error: \", e"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "v7_aT7Hr5rnC",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The easiest way to initialize a variable is to call `global_variables_initializer`. Note the use of `Session.run()`, which is roughly equivalent to `eval()`."
      ]
    },
    {
      "metadata": {
        "id": "z2lvhrxI5zJF",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "with g.as_default():\n",
        "  with tf.Session() as sess:\n",
        "    initialization = tf.global_variables_initializer()\n",
        "    sess.run(initialization)\n",
        "    # Now, variables can be accessed normally, and have values assigned to them.\n",
        "    print v.eval()\n",
        "    print w.eval()\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0GkYh7jf6JUd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Once initialized, variables will maintain their value within the same session (however, when starting a new session, you will need to re-initialize them):"
      ]
    },
    {
      "metadata": {
        "id": "_E8_lhS06IoV",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "with g.as_default():\n",
        "  with tf.Session() as sess:\n",
        "    sess.run(tf.global_variables_initializer())\n",
        "    # These three prints will print the same value.\n",
        "    print w.eval()\n",
        "    print w.eval()\n",
        "    print w.eval()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "yrZ31hPw66uy",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "To change the value of a variable, use the `assign` op. Note that simply creating the `assign` op will not have any effect. As with initialization, you have to `run` the assignment op to update the variable value:"
      ]
    },
    {
      "metadata": {
        "id": "zD0D1DCR7NBX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "with g.as_default():\n",
        "  with tf.Session() as sess:\n",
        "    sess.run(tf.global_variables_initializer())\n",
        "    # This should print the variable's initial value.\n",
        "    print v.eval()\n",
        "\n",
        "    assignment = tf.assign(v, [7])\n",
        "    # The variable has not been changed yet!\n",
        "    print v.eval()\n",
        "\n",
        "    # Execute the assignment op.\n",
        "    sess.run(assignment)\n",
        "    # Now the variable is updated.\n",
        "    print v.eval()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "tB78Zq4h78Zr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "There are many more topics about variables that we didn't cover here, such as loading and storing. To learn more, see the [TensorFlow docs](https://www.tensorflow.org/programmers_guide/variables)."
      ]
    },
    {
      "metadata": {
        "id": "iFIOcnfz_Oqw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Exercise #2: Simulate 10 rolls of two dice.\n",
        "\n",
        "Create a dice simulation, which generates a `10x3` 2-D tensor in which:\n",
        "\n",
        "  * Columns `1` and `2` each hold one throw of one six-sided die (with values 1–6).\n",
        "  * Column `3` holds the sum of Columns `1` and `2` on the same row.\n",
        "\n",
        "For example, the first row might have the following values:\n",
        "\n",
        "  * Column `1` holds `4`\n",
        "  * Column `2` holds `3`\n",
        "  * Column `3` holds `7`\n",
        "\n",
        "You'll need to explore the [TensorFlow documentation](https://www.tensorflow.org/api_guides/python/array_ops) to solve this task."
      ]
    },
    {
      "metadata": {
        "id": "ocwT0iXH-nhT",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Write your code for Task 2 here."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Kt7aojXkR_qS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Solution\n",
        "\n",
        "Click below for a solution."
      ]
    },
    {
      "metadata": {
        "id": "6UUluecQSCvr",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "with tf.Graph().as_default(), tf.Session() as sess:\n",
        "  # Task 2: Simulate 10 throws of two six-sided dice. Store the results\n",
        "  # in a 10x3 matrix.\n",
        "\n",
        "  # We're going to place dice throws inside two separate\n",
        "  # 10x1 matrices. We could have placed dice throws inside\n",
        "  # a single 10x2 matrix, but adding different columns of\n",
        "  # the same matrix is tricky. We also could have placed\n",
        "  # dice throws inside two 1-D tensors (vectors); doing so\n",
        "  # would require transposing the result.\n",
        "  dice1 = tf.Variable(tf.random_uniform([10, 1],\n",
        "                                        minval=1, maxval=7,\n",
        "                                        dtype=tf.int32))\n",
        "  dice2 = tf.Variable(tf.random_uniform([10, 1],\n",
        "                                        minval=1, maxval=7,\n",
        "                                        dtype=tf.int32))\n",
        "\n",
        "  # We may add dice1 and dice2 since they share the same shape\n",
        "  # and size.\n",
        "  dice_sum = tf.add(dice1, dice2)\n",
        "\n",
        "  # We've got three separate 10x1 matrices. To produce a single\n",
        "  # 10x3 matrix, we'll concatenate them along dimension 1.\n",
        "  resulting_matrix = tf.concat(\n",
        "      values=[dice1, dice2, dice_sum], axis=1)\n",
        "\n",
        "  # The variables haven't been initialized within the graph yet,\n",
        "  # so let's remedy that.\n",
        "  sess.run(tf.global_variables_initializer())\n",
        "\n",
        "  print(resulting_matrix.eval())"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}