2-Functions-External_Libraries-File_IO.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ISRC Python Workshop: Baiscs II\n",
    "\n",
    "__Functions, File I/O and External Libraries__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "@author: Zhiya Zuo\n",
    "\n",
    "@email: zhiya-zuo@uiowa.edu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calling functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Previously, we have already made use of many built-in functions to facilitate programming. Function is a block of codes with input arguments (and, optionally, return values) for specific purposes. In Python ( and many other languages), a function call is as the following:\n",
    "\n",
    "```python\n",
    ">> output = function(input_argument)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:47:37.407545Z",
     "start_time": "2018-10-25T17:47:37.389644Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "range(0, 5)"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "range(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that Python 3 use [`iterator`](https://stackoverflow.com/questions/25653996/what-is-the-difference-between-list-and-iterator-in-python) for 'range' function, we can manually convert the output into `list` so that we can see the output explicitly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:48:25.160955Z",
     "start_time": "2018-10-25T17:48:25.155505Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2, 3, 4]"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(range(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As another example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:48:45.482719Z",
     "start_time": "2018-10-25T17:48:45.476899Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.5"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "abs(-3.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In many cases we need more sophisticated usage of functions, where we need to use more than one input arguments. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:20:29.734990Z",
     "start_time": "2018-10-25T17:20:29.729907Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5, 4, 3, 2, 1]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(range(5, 0, -1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A second example, sort a dictionary by values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:50:16.189583Z",
     "start_time": "2018-10-25T17:50:16.183753Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'b', 'c']"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = {'a': 100, 'c': 50, 'b': 70}\n",
    "sorted(d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:53:27.808548Z",
     "start_time": "2018-10-25T17:53:27.798150Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a': 100, 'c': 50, 'b': 70}"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:53:33.866813Z",
     "start_time": "2018-10-25T17:53:33.861638Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d['a']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:51:39.294194Z",
     "start_time": "2018-10-25T17:51:39.286761Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['c', 'b', 'a']"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(d, key=lambda k: d[k])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lambda functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aha, we just saw something different: `lambda`!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lambda functions are just functions, except that they are anonymous (literally). See [here](https://stackoverflow.com/questions/890128/why-are-python-lambdas-useful) for many good discussions. In short, you can use regular functions to achieve anything with `lambda`. Yet, it is handy because it is lightweight and anonymous."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The example above is actually a good example of when to use `lambda`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:20:29.761643Z",
     "start_time": "2018-10-25T17:20:29.757012Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['c', 'b', 'a']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(d, key=lambda k: d[k])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is one and only one expression within the `lambda` function. In this case, the input is `k`, a key inside the dictionary `d` and the output is `d[k]`, the value in `d` w.r.t. the key `k`. Therefore we are sorting our dictionary keys by their values instead of the keys themselves."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define our own functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we are not limited to built-in functions only. Let's now try make our own functions. Before that, we need to be clear on the structure of a function\n",
    "```python\n",
    "def func_name(arg1, arg2, arg3, ...):\n",
    "    #####################\n",
    "    # Do something here #\n",
    "    #####################\n",
    "    return output\n",
    "```\n",
    "\n",
    "\\* *`return output` is NOT required*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following example, we make use of `sum`, a built-in function to sum up numeric iterables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:00:12.954116Z",
     "start_time": "2018-10-25T18:00:12.950910Z"
    }
   },
   "outputs": [],
   "source": [
    "def mySum(list_to_sum):\n",
    "    return sum(list_to_sum)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:00:13.219092Z",
     "start_time": "2018-10-25T18:00:13.214135Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mySum(range(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A more complicated one that does not use `sum` function.\n",
    "- Do not remember for loop? Check out [here](https://github.com/zhiyzuo/python-tutorial/blob/master/1-Variables-Data_Structures-Control_Logic.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:00:55.632800Z",
     "start_time": "2018-10-25T18:00:55.628985Z"
    }
   },
   "outputs": [],
   "source": [
    "def mySumUsingLoop(list_to_sum):\n",
    "    sum_ = list_to_sum[0]\n",
    "    for item in list_to_sum[1:]:\n",
    "        sum_ += item\n",
    "    return sum_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:00:55.894819Z",
     "start_time": "2018-10-25T18:00:55.887776Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mySumUsingLoop(range(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*The two example functions are not doing anything interesting but just served as illustrations to build customized functions.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's see how we can sort a dictionary by values using functions instead of `lambda`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:01:10.928528Z",
     "start_time": "2018-10-25T18:01:10.923553Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a': 100, 'c': 50, 'b': 70}"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:01:11.516168Z",
     "start_time": "2018-10-25T18:01:11.512844Z"
    }
   },
   "outputs": [],
   "source": [
    "def my_key(key):\n",
    "    return d[key]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:01:30.640489Z",
     "start_time": "2018-10-25T18:01:30.634567Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['c', 'b', 'a']"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(d, key=my_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See, `lambda` is way simpler than defining a function explicitly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FIle I/O"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section is about some basics on reading and writing data, in Python native style"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Write data to a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:04:05.330388Z",
     "start_time": "2018-10-25T18:04:05.317270Z"
    }
   },
   "outputs": [],
   "source": [
    "f = open(\"tmp1.csv\", \"w\") # f is a file handler, while \"w\" is the mode (w for write)\n",
    "for item in range(6):\n",
    "    f.write(str(item))\n",
    "    # add newline character \n",
    "    f.write(\"\\n\") \n",
    "    # alternatively, we can do:\n",
    "    # f.write(str(item)+\"\\n\") because we can concat two strings by using `+`\n",
    "f.close() # close the filer handler for security reasons."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "check out the file we just created `tmp.csv`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:04:14.746566Z",
     "start_time": "2018-10-25T18:04:14.613060Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\r\n",
      "1\r\n",
      "2\r\n",
      "3\r\n",
      "4\r\n",
      "5\r\n"
     ]
    }
   ],
   "source": [
    "cat tmp1.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that without the typecasting from `int` to `str`, an error will be raised."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A more commonly used way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:05:39.217600Z",
     "start_time": "2018-10-25T18:05:39.211303Z"
    }
   },
   "outputs": [],
   "source": [
    "with open(\"tmp2.csv\", \"w\") as f: # f is a file handler, while \"w\" is the mode (w for write)\n",
    "    for item in range(4):\n",
    "        f.write(str(item))\n",
    "        f.write(\"\\n\") # add newline character"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:05:39.624751Z",
     "start_time": "2018-10-25T18:05:39.496854Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\r\n",
      "1\r\n",
      "2\r\n",
      "3\r\n"
     ]
    }
   ],
   "source": [
    "cat tmp2.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No need to close because of `with`.\n",
    "\n",
    "See more here:\n",
    "1. https://stackoverflow.com/questions/3012488/what-is-the-python-with-statement-designed-for\n",
    "2. https://docs.python.org/3/whatsnew/2.6.html#pep-343-the-with-statement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Occasionally, we need to _append new elements_ instead of _overwriting_ existing files. In this case, we should use `a` mode in our `open` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:06:09.439732Z",
     "start_time": "2018-10-25T18:06:09.434341Z"
    }
   },
   "outputs": [],
   "source": [
    "with open(\"tmp2.csv\", \"a\") as f:\n",
    "    for item in range(15, 19):\n",
    "        f.write(str(item)+\"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:06:09.845265Z",
     "start_time": "2018-10-25T18:06:09.718758Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\r\n",
      "1\r\n",
      "2\r\n",
      "3\r\n",
      "15\r\n",
      "16\r\n",
      "17\r\n",
      "18\r\n"
     ]
    }
   ],
   "source": [
    "cat tmp2.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Read data to a file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To read a text file into Python, we use `r` mode (for _read_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:08:01.936428Z",
     "start_time": "2018-10-25T18:08:01.931565Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['0\\n', '1\\n', '2\\n', '3\\n', '4\\n', '5\\n']\n"
     ]
    }
   ],
   "source": [
    "f = open(\"tmp1.csv\", \"r\") # this time, use read mode\n",
    "contents = [item for item in f] # list comprehension. This is the same as for-loop but more concise\n",
    "print(contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Usually, we do not like trailing newlines. We can use `strip` to remove them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:08:02.351111Z",
     "start_time": "2018-10-25T18:08:02.341732Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['0', '1', '2', '3', '4', '5']\n"
     ]
    }
   ],
   "source": [
    "contents = [item.strip(\"\\n\") for item in contents] # strip the newline\n",
    "print(contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`map` is a function to do similar things like _list comprehension_. See [here](https://stackoverflow.com/questions/10973766/understanding-the-map-function) for more discussions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:08:45.960510Z",
     "start_time": "2018-10-25T18:08:45.956406Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1, 2, 3, 4, 5]\n"
     ]
    }
   ],
   "source": [
    "int_values = list(map(int, contents)) # map the values into integer type\n",
    "print(int_values)\n",
    "f.close() # always remember to close the file handler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also using with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:09:06.950061Z",
     "start_time": "2018-10-25T18:09:06.943884Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before converting to `int`\n",
      "['0', '1', '2', '3', '4', '5']\n",
      "After...\n",
      "[0, 1, 2, 3, 4, 5]\n"
     ]
    }
   ],
   "source": [
    "with open(\"tmp1.csv\", \"r\") as f:\n",
    "    contents = [item for item in f] # list comprehension. This is the same as for-loop but more concise\n",
    "    contents = [item.strip(\"\\n\") for item in contents] # strip the newline\n",
    "    print('Before converting to `int`')\n",
    "    print(contents)\n",
    "    int_values = list(map(int, contents)) # map the values into integer type\n",
    "    print('After...')\n",
    "    print(int_values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often times, we need either internal or external help for complicated computation tasks. In these occasions, we need to _import libraries_. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Built-in libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python provides many built-in packages to prevent extra work on some common and useful functions\n",
    "\n",
    "We will use __math__ as an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:11:30.413987Z",
     "start_time": "2018-10-25T18:11:30.411351Z"
    }
   },
   "outputs": [],
   "source": [
    "import math # use import to load a library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use functions from the library, do: `library_name.function_name`. For example, when we want to calculate the logarithm using a function from `math` library, we can do `math.log`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:12:36.793378Z",
     "start_time": "2018-10-25T18:12:36.788201Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "e^x = e^3 = 20.085537\n",
      "log(x) = log(3) = 1.098612\n"
     ]
    }
   ],
   "source": [
    "x = 3\n",
    "print(\"e^x = e^3 = %f\"%math.exp(x))\n",
    "print(\"log(x) = log(3) = %f\"%math.log(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also import one specific function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:12:37.162731Z",
     "start_time": "2018-10-25T18:12:37.157103Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20.085536923187668\n"
     ]
    }
   ],
   "source": [
    "from math import exp # You can import a specific function\n",
    "print(exp(x)) # This way, you don't need to use math.exp but just exp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or all:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:20:30.324507Z",
     "start_time": "2018-10-25T17:20:30.320936Z"
    }
   },
   "outputs": [],
   "source": [
    "from math import * # Import all functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:20:30.331795Z",
     "start_time": "2018-10-25T17:20:30.327483Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20.085536923187668\n",
      "1.0986122886681098\n"
     ]
    }
   ],
   "source": [
    "print(exp(x))\n",
    "print(log(x)) # Before importing math, calling `exp` or `log` will raise errors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Depending on what you want to achieve, you may want to choose between importing a few or all (by `*`) functions within a package."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### External libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-10-21T16:45:05.266743Z",
     "start_time": "2017-10-21T16:45:05.260803Z"
    }
   },
   "source": [
    "There are times you'll want some advanced utility functions not provided by Python. There are many useful packages by developers.\n",
    "\n",
    "We'll use __numpy__ as an example. (__numpy__, __scipy__, __matplotlib__,and probably __pandas__ will be of the most importance to you for data analyses.\n",
    "\n",
    "Installation of packages for Python is the easiest using <a href=\"https://packaging.python.org/installing/\" target=\"_blank\">pip</a>:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "~$ pip install numpy scipy pandas\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you use Anaconda, I beleive all these are ready for your use."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading external libraries is just the same as built-in ones. To use _alias_ for easier access to the libraries, we can import a library by: `import library_long_name as short_name`. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:13:42.776598Z",
     "start_time": "2018-10-25T18:13:42.770831Z"
    }
   },
   "outputs": [],
   "source": [
    "# After you install numpy, load it\n",
    "import numpy as np # you can use np instead of numpy to call the functions in numpy package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:15:33.255909Z",
     "start_time": "2018-10-25T18:15:33.250898Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1. 2. 3.]\n",
      " [4. 5. 7.]]\n",
      "<class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "source": [
    "x = np.array([[1,2,3], [4,5,7]], dtype=np.float) # create a numpy array object, specify the data type as float\n",
    "print(x)\n",
    "print(type(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can call `shape` function designed for `numpy.ndarray` class to check the dimension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:15:44.041903Z",
     "start_time": "2018-10-25T18:15:44.036465Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 3)"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike `list`, we have to use one single data type for all elements in an array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:16:19.660627Z",
     "start_time": "2018-10-25T18:16:19.655016Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['1', 'yes'], dtype='<U21')"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = np.array([1, 'yes'])\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:16:34.203322Z",
     "start_time": "2018-10-25T18:16:34.195223Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('1', numpy.str_)"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y[0], type(y[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:16:49.525338Z",
     "start_time": "2018-10-25T18:16:49.519782Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, int)"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_list = [1, 'yes']\n",
    "y_list[0], type(y_list[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Scipy/Numpy__ provides extensive utilities to manipulate data and simple analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:17:50.704195Z",
     "start_time": "2018-10-25T18:17:50.701194Z"
    }
   },
   "outputs": [],
   "source": [
    "from scipy.stats import pearsonr, spearmanr # correlation functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:17:50.766246Z",
     "start_time": "2018-10-25T18:17:50.743057Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0.9819805060619659, 0.1210377183236763)\n",
      "SpearmanrResult(correlation=1.0, pvalue=0.0)\n"
     ]
    }
   ],
   "source": [
    "print(pearsonr(x[1, :], x[0, :]))\n",
    "print(spearmanr(x[1, :], x[0, :]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Pandas__ (Python Data Analysis Library) is a great package for data structures: `DataFrame`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you're familar with `R`, then you must love `pandas.DataFrame` data structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:18:13.027876Z",
     "start_time": "2018-10-25T18:18:13.025121Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:18:55.978258Z",
     "start_time": "2018-10-25T18:18:55.969575Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 2., 3.],\n",
       "       [4., 5., 7.]])"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:19:06.351817Z",
     "start_time": "2018-10-25T18:19:06.337655Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0    1    2\n",
       "0  1.0  2.0  3.0\n",
       "1  4.0  5.0  7.0"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_df = pd.DataFrame(x)\n",
    "x_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Easy import/export"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:20:32.720410Z",
     "start_time": "2018-10-25T17:20:32.701648Z"
    }
   },
   "outputs": [],
   "source": [
    "x_df.to_csv('tmp_pd.csv', index=False) # `index=False`: do not write row indices to file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:20:32.774467Z",
     "start_time": "2018-10-25T17:20:32.755799Z"
    }
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('tmp_pd.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:20:32.816525Z",
     "start_time": "2018-10-25T17:20:32.781308Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0    1    2\n",
       "0  1.0  2.0  3.0\n",
       "1  4.0  5.0  6.0"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quick Intro to Numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of using the native data structures, we use `numpy.ndarray` for data analytics most of the time. While they are not as \"flexible\" as lists, they are easy to use and have better performance. As Numpy's official documentation states:\n",
    "> NumPy’s main object is the homogeneous multidimensional array. It is a table of elements (usually numbers), all of the same type, indexed by a tuple of positive integers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we were using it just now, the most common alias for `numpy` is `np`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:20:55.886804Z",
     "start_time": "2018-10-25T18:20:55.883788Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Depending on what types of analyses we are going to work on later, the most appropriate array initialization methods can be choosed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### By hand"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is very similar to creating a list of elements manually, except that we wrap the list around by `np.array()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:21:09.714465Z",
     "start_time": "2018-10-25T18:21:09.708143Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 2, 3, 8])"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr = np.array([1,2,3,8])\n",
    "arr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:21:10.376843Z",
     "start_time": "2018-10-25T18:21:10.372139Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4,)"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Multidimensional arrays: seperated by comma"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1 by 4: 1 row and 4 columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:22:13.908319Z",
     "start_time": "2018-10-25T18:22:13.903647Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 4)"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr = np.array([[1,2,3,8]])\n",
    "arr.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:22:30.154253Z",
     "start_time": "2018-10-25T18:22:30.148702Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2, 3, 8]])"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3 by 4: 3 row and 4 columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:22:47.168037Z",
     "start_time": "2018-10-25T18:22:47.162787Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 4)"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr = np.array([[1,2,3,8], [3,2,3,2], [4,5,0,8]])\n",
    "arr.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:22:47.618800Z",
     "start_time": "2018-10-25T18:22:47.613359Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2, 3, 8],\n",
       "       [3, 2, 3, 2],\n",
       "       [4, 5, 0, 8]])"
      ]
     },
     "execution_count": 145,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### By functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many special array initialization methods to call:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:23:26.355528Z",
     "start_time": "2018-10-25T18:23:26.350971Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0]])"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.zeros([3,5], dtype=int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:20:32.915771Z",
     "start_time": "2018-10-25T17:20:32.908663Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 1., 1., 1., 1.],\n",
       "       [1., 1., 1., 1., 1.],\n",
       "       [1., 1., 1., 1., 1.]])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.ones([3,5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:20:32.928709Z",
     "start_time": "2018-10-25T17:20:32.919025Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 0., 0.],\n",
       "       [0., 1., 0.],\n",
       "       [0., 0., 1.]])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.eye(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Arithmatic operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The rules are very similar to R: they are generally element wise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:24:37.986303Z",
     "start_time": "2018-10-25T18:24:37.981265Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2, 3, 8],\n",
       "       [3, 2, 3, 2],\n",
       "       [4, 5, 0, 8]])"
      ]
     },
     "execution_count": 147,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:24:38.430440Z",
     "start_time": "2018-10-25T18:24:38.411917Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 6, 12, 18, 48],\n",
       "       [18, 12, 18, 12],\n",
       "       [24, 30,  0, 48]])"
      ]
     },
     "execution_count": 148,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr * 6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:20:32.958899Z",
     "start_time": "2018-10-25T17:20:32.951229Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-4, -3, -2,  3],\n",
       "       [-2, -3, -2, -3],\n",
       "       [-1,  0, -5,  3]])"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr - 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:20:32.970030Z",
     "start_time": "2018-10-25T17:20:32.963074Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2.71828183e+00, 7.38905610e+00, 2.00855369e+01, 2.98095799e+03],\n",
       "       [2.00855369e+01, 7.38905610e+00, 2.00855369e+01, 7.38905610e+00],\n",
       "       [5.45981500e+01, 1.48413159e+02, 1.00000000e+00, 2.98095799e+03]])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.exp(arr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that if we want conduct matrix multiplication, we need to use `@` or `.dot` function, since `*` still means element wise computation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:20:32.981988Z",
     "start_time": "2018-10-25T17:20:32.973813Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1],\n",
       "       [3],\n",
       "       [2],\n",
       "       [0]])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr_2 = np.array([[1], [3], [2], [0]])\n",
    "arr_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:20:33.001842Z",
     "start_time": "2018-10-25T17:20:32.985348Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[13],\n",
       "       [15],\n",
       "       [19]])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr @ arr_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:20:33.017275Z",
     "start_time": "2018-10-25T17:20:33.008780Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[13],\n",
       "       [15],\n",
       "       [19]])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr.dot(arr_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Operation based on itself"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many class methods to calculate some statistics of the array itself along some axis:\n",
    "- `axis=1` means row-wise\n",
    "- `axis=0` means column-wise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:20:33.031146Z",
     "start_time": "2018-10-25T17:20:33.020661Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2, 3, 8],\n",
       "       [3, 2, 3, 2],\n",
       "       [4, 5, 0, 8]])"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:20:33.048518Z",
     "start_time": "2018-10-25T17:20:33.033767Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:20:33.060765Z",
     "start_time": "2018-10-25T17:20:33.051750Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([8, 3, 8])"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr.max(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T17:20:33.072580Z",
     "start_time": "2018-10-25T17:20:33.064562Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4, 5, 3, 8])"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr.max(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:28:06.467191Z",
     "start_time": "2018-10-25T18:28:06.458802Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1,  3,  6, 14, 17, 19, 22, 24, 28, 33, 33, 41])"
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr.cumsum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:28:09.052626Z",
     "start_time": "2018-10-25T18:28:09.047583Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1,  3,  6, 14],\n",
       "       [ 3,  5,  8, 10],\n",
       "       [ 4,  9,  9, 17]])"
      ]
     },
     "execution_count": 151,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr.cumsum(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Indexing and slicing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most important part is how to index and slice a `np.array`. It is actually very similar to `list`, except that we now may have more index elements because there are more than one dimension for most of the datasets in real life"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1 dimensional case"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:28:39.875850Z",
     "start_time": "2018-10-25T18:28:39.862936Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  1,   2,   8, 100])"
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a1 = np.array([1,2,8,100])\n",
    "a1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:28:40.194494Z",
     "start_time": "2018-10-25T18:28:40.187561Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a1[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:28:54.262749Z",
     "start_time": "2018-10-25T18:28:54.255392Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a1[-2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:29:03.024941Z",
     "start_time": "2018-10-25T18:29:03.019955Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  1,   2, 100])"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a1[[0,1,3]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use boolean values to index\n",
    "- `True` means we want this element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:29:28.874057Z",
     "start_time": "2018-10-25T18:29:28.869300Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False, False,  True,  True])"
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a1 > 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:30:00.091948Z",
     "start_time": "2018-10-25T18:30:00.086799Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  8, 100])"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a1[a1 > 3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 2 dimensional case"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:30:01.277543Z",
     "start_time": "2018-10-25T18:30:01.270262Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2, 3, 8],\n",
       "       [3, 2, 3, 2],\n",
       "       [4, 5, 0, 8]])"
      ]
     },
     "execution_count": 160,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using only one number to index will lead to a subset of the original multidimenional array: also an array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:30:15.881928Z",
     "start_time": "2018-10-25T18:30:15.877093Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 2, 3, 8])"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:30:21.305735Z",
     "start_time": "2018-10-25T18:30:21.299368Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 162,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(arr[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we have 2 dimensions now, there are 2 indices we can use for indexing the 2 dimensions respectively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:35:00.665066Z",
     "start_time": "2018-10-25T18:35:00.661163Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 184,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr[0,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `:` to indicate everything along that axis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:35:01.023310Z",
     "start_time": "2018-10-25T18:35:01.012994Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3, 2, 3, 2])"
      ]
     },
     "execution_count": 185,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:35:01.286832Z",
     "start_time": "2018-10-25T18:35:01.277934Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3, 2, 3, 2])"
      ]
     },
     "execution_count": 186,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr[1, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:35:01.415490Z",
     "start_time": "2018-10-25T18:35:01.405283Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True,  True,  True,  True])"
      ]
     },
     "execution_count": 187,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr[1,:] == arr[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:35:01.582330Z",
     "start_time": "2018-10-25T18:35:01.573495Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2, 2, 5])"
      ]
     },
     "execution_count": 188,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr[:, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 3 dimensional case"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a final example, we look at a 3d array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:35:02.101149Z",
     "start_time": "2018-10-25T18:35:02.092391Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([69,  5, 51, 67, 35, 20,  8, 40, 66, 94,  0, 25, 64, 78, 47, 48, 98,\n",
       "       59, 86, 82, 67, 63, 81, 34])"
      ]
     },
     "execution_count": 189,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr_3 = np.random.randint(low=0, high=100, size=24)\n",
    "arr_3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use [`reshape`](https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.reshape.html) to manipulate the shape of an array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:35:02.489922Z",
     "start_time": "2018-10-25T18:35:02.478082Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[69,  5],\n",
       "        [51, 67],\n",
       "        [35, 20],\n",
       "        [ 8, 40]],\n",
       "\n",
       "       [[66, 94],\n",
       "        [ 0, 25],\n",
       "        [64, 78],\n",
       "        [47, 48]],\n",
       "\n",
       "       [[98, 59],\n",
       "        [86, 82],\n",
       "        [67, 63],\n",
       "        [81, 34]]])"
      ]
     },
     "execution_count": 190,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr_3 = arr_3.reshape(3,4,2)\n",
    "arr_3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:35:02.729883Z",
     "start_time": "2018-10-25T18:35:02.725296Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[69,  5],\n",
       "       [51, 67],\n",
       "       [35, 20],\n",
       "       [ 8, 40]])"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr_3[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:35:02.850093Z",
     "start_time": "2018-10-25T18:35:02.843697Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([40, 48, 34])"
      ]
     },
     "execution_count": 192,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr_3[:, 3, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-25T18:35:03.048599Z",
     "start_time": "2018-10-25T18:35:03.040467Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "34"
      ]
     },
     "execution_count": 193,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr_3[2, 3, 1]"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "840px",
    "left": "0px",
    "right": "1111px",
    "top": "113px",
    "width": "253px"
   },
   "toc_section_display": "block",
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}