Session 4.ipynb

{
 "metadata": {
  "name": "Session 4"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Don't reinvent the wheel\n",
      "\n",
      "It's possible to use the standard Python data structures for building blocks for most structures you could imagine.\n",
      "\n",
      "- `list`\n",
      "- `tuple`\n",
      "- `set`\n",
      "- `dict`\n",
      "\n",
      "## Collections\n",
      "\n",
      "There is a python module `collections` which contain data structures many find themselves implement often\n",
      "\n",
      "- `Counter`\n",
      "- `defaultdict`\n",
      "- `OrderedDict`\n",
      "- `deque`\n",
      "- `namedtuple()`"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Hashable objects\n",
      "\n",
      "A _hashable collection_ in Python is any data structure which stores data in such a way that it uses a hash table for data accession.\n",
      "\n",
      "To be placeable in a hashable collection an object need to implement `__hash__()`, which should return an `int`, which should also never change value during the lifetime of that object. Otherwise the object would be stored in the wrong place after it is changed.\n",
      "\n",
      "In addition, the objects need to implement `__eq__()` or `__cmp__()` so it can be evaluated whether two colliding objects are different from each other or if it is the same object.\n",
      "\n",
      "Objects satisfying these two conditions are called _hashable_.\n",
      "\n",
      "All the immutable built in Python data types are hashable (i.e. the primitive data types, as well as `str` and `tuple`)."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### `Counter`\n",
      "\n",
      "Counting the number of occurrences of items is very common, and can in Python be implemented with a dictionary where the keys are the objects being counted, and the values the count.\n",
      "\n",
      "An efficient implementation of this, along with many convenient features, is avalable in `collections.Counter`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from collections import Counter"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "c = Counter()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "c"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 3,
       "text": [
        "Counter()"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "c[\"a\"]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 4,
       "text": [
        "0"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "c.update(\"a\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "c"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 8,
       "text": [
        "Counter({'a': 2})"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We can also instantiate the `Counter` by passing it a container of hashable objects."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "c = Counter(\"A quite short but in other ways normal English sentence\".lower().replace(\" \", \"\"))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "c"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 10,
       "text": [
        "Counter({'e': 6, 'n': 5, 't': 5, 's': 4, 'a': 3, 'i': 3, 'h': 3, 'o': 3, 'r': 3, 'l': 2, 'u': 2, 'c': 1, 'b': 1, 'g': 1, 'm': 1, 'q': 1, 'w': 1, 'y': 1})"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "c.most_common(5)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 11,
       "text": [
        "[('e', 6), ('n', 5), ('t', 5), ('s', 4), ('a', 3)]"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "c."
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 13,
       "text": [
        "6"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "`Counters` supports mathematical operations which makes them behave like multisets, for which there are some algorithmic uses."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### `defaultdict`\n",
      "\n",
      "Say that we want to implement the `Counter` functionality for a normal dict ourselves. A function taking a container of hashables and returning a `dict` of counts would look like this."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def count_items(collection):\n",
      "    counts = dict()\n",
      "    for item in iter(collection):\n",
      "        if item in counts:\n",
      "            counts[item] += 1\n",
      "        else:\n",
      "            counts[item] = 1\n",
      "    \n",
      "    return counts"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 14
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "count_items(\"test\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 15,
       "text": [
        "{'e': 1, 's': 1, 't': 2}"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This recipe of doing things to items in a `dict` in case the exist, or adding them if they don't, is _extremely_ common. So in the interest of not reinventing the wheel there is `collections.defauldict`.\n",
      "\n",
      "A `defaultdict` is instantiated with a _factory_ for objects which will be added to the dictionary if we try to access an item which is not there.\n",
      "\n",
      "The term _factory_ refers to an object which creates other objects.\n",
      "\n",
      "For example, `int` is a `type` object, which produces `int`s when called."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "type(int)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 17,
       "text": [
        "type"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "type(int())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 18,
       "text": [
        "int"
       ]
      }
     ],
     "prompt_number": 18
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "int()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 20,
       "text": [
        "0"
       ]
      }
     ],
     "prompt_number": 20
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Normally types are used as factories, but it can actually be any function which returns something."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from collections import defaultdict\n",
      "\n",
      "A = defaultdict(int)\n",
      "A"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 16,
       "text": [
        "defaultdict(<type 'int'>, {})"
       ]
      }
     ],
     "prompt_number": 16
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "A[\"test\"]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 21,
       "text": [
        "0"
       ]
      }
     ],
     "prompt_number": 21
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "A"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 22,
       "text": [
        "defaultdict(<type 'int'>, {'test': 0})"
       ]
      }
     ],
     "prompt_number": 22
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "B = defaultdict(list)\n",
      "B[\"one\"].append(5)\n",
      "B"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 23,
       "text": [
        "defaultdict(<type 'list'>, {'one': [5]})"
       ]
      }
     ],
     "prompt_number": 23
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "B[\"one\"].append(3)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 24
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "B"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 25,
       "text": [
        "defaultdict(<type 'list'>, {'one': [5, 3]})"
       ]
      }
     ],
     "prompt_number": 25
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "C = defaultdict(lambda: defaultdict(int))\n",
      "C[\"outer\"][\"inner\"] += 7\n",
      "C"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 26,
       "text": [
        "defaultdict(<function <lambda> at 0x10fd3ecf8>, {'outer': defaultdict(<type 'int'>, {'inner': 7})})"
       ]
      }
     ],
     "prompt_number": 26
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def defaultdict_factory():\n",
      "    return defaultdict(defaultdict_factory)\n",
      "\n",
      "D = defaultdict(defaultdict_factory)\n",
      "D[0][0][0][0][0] = \"Inside a dict of dicts of ... of dicts (dynamically)\"\n",
      "D"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 27,
       "text": [
        "defaultdict(<function defaultdict_factory at 0x10fd3ed70>, {0: defaultdict(<function defaultdict_factory at 0x10fd3ed70>, {0: defaultdict(<function defaultdict_factory at 0x10fd3ed70>, {0: defaultdict(<function defaultdict_factory at 0x10fd3ed70>, {0: defaultdict(<function defaultdict_factory at 0x10fd3ed70>, {0: 'Inside a dict of dicts of ... of dicts (dynamically)'})})})})})"
       ]
      }
     ],
     "prompt_number": 27
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### `OrderedDict`\n",
      "\n",
      "Since `dict`s are implemented to give fast access to the items, the data is stored in a way that makes that quick. But sometimes you might want it sorted so it is possible to iterate over it in a predictable way."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "d = {}\n",
      "\n",
      "for k, v in zip([\"a\", \"b\", \"g\", \"d\"], [u\"\u03b1\", u\"\u03b2\", u\"\u03b3\", u\"\u03b4\"]):\n",
      "    d[k] = v\n",
      "\n",
      "d"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 28,
       "text": [
        "{'a': u'\\u03b1', 'b': u'\\u03b2', 'd': u'\\u03b4', 'g': u'\\u03b3'}"
       ]
      }
     ],
     "prompt_number": 28
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from collections import OrderedDict\n",
      "\n",
      "od = OrderedDict()\n",
      "\n",
      "for k, v in zip([\"a\", \"b\", \"g\", \"d\"], [u\"\u03b1\", u\"\u03b2\", u\"\u03b3\", u\"\u03b4\"]):\n",
      "    od[k] = v\n",
      "\n",
      "od"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 29,
       "text": [
        "OrderedDict([('a', u'\\u03b1'), ('b', u'\\u03b2'), ('g', u'\\u03b3'), ('d', u'\\u03b4')])"
       ]
      }
     ],
     "prompt_number": 29
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "\"a\", u\"a\", r\"a\""
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 30,
       "text": [
        "('a', u'a', 'a')"
       ]
      }
     ],
     "prompt_number": 30
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### `deque`\n",
      "\n",
      "In Python, `list`s are optimized for operations on a fixed size (even though the sizes of the _elements_ might vary, unlike with an `array`)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "a = range(5)\n",
      "a"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 31,
       "text": [
        "[0, 1, 2, 3, 4]"
       ]
      }
     ],
     "prompt_number": 31
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "When treating a `list` like a stack (where the right end is the top), everything is fine"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "a.pop()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 32,
       "text": [
        "4"
       ]
      }
     ],
     "prompt_number": 32
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "a"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 33,
       "text": [
        "[0, 1, 2, 3]"
       ]
      }
     ],
     "prompt_number": 33
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "a.append(6)\n",
      "a.append(7)\n",
      "a"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 34,
       "text": [
        "[0, 1, 2, 3, 6, 7]"
       ]
      }
     ],
     "prompt_number": 34
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "a.pop()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 35,
       "text": [
        "7"
       ]
      }
     ],
     "prompt_number": 35
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "But when working from the left end will cause `list` performance to be $O(n)$ rather than $O(1)$ due to memory movement."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "a.pop(0)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 36,
       "text": [
        "0"
       ]
      }
     ],
     "prompt_number": 36
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "a.insert(0, 8)\n",
      "a.insert(0, 9)\n",
      "a"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 37,
       "text": [
        "[9, 8, 1, 2, 3, 6]"
       ]
      }
     ],
     "prompt_number": 37
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "A _`deque`_ (\"double-ended queue\") is a structure which gives $O(1)$ time for both insertion and accession at both ends.\n",
      "\n",
      "What you're giving up is the ability to slice.\n",
      "\n",
      "Accession by indexes is near $O(1)$ close to the ends, but goes towards $O(n)$ closer to the middle.\n",
      "\n",
      "One can view a `deque` as a circular `list` where we have a reference to a particular section of the circle.\n",
      "\n",
      "By essentially popping from one end and inserting to the other, the `deque` can be rotated. This is something that can be exploited when writing algorithms where we can take advantage of data locality for portions of the runtime."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from collections import deque"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 38
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%timeit\n",
      "# Right append - pop list\n",
      "\n",
      "l = []\n",
      "for z in xrange(1000):\n",
      "    l.append(z)\n",
      "\n",
      "for y in xrange(len(l)):\n",
      "    l.pop()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1000 loops, best of 3: 434 us per loop\n"
       ]
      }
     ],
     "prompt_number": 39
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%timeit\n",
      "# Right append - pop deque\n",
      "\n",
      "q = deque()\n",
      "for z in xrange(1000):\n",
      "   q.append(z)\n",
      "\n",
      "while True:\n",
      "    try:\n",
      "        q.pop()\n",
      "    except:\n",
      "        break"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1000 loops, best of 3: 455 us per loop\n"
       ]
      }
     ],
     "prompt_number": 40
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%timeit\n",
      "# Left append - pop list\n",
      "\n",
      "l = []\n",
      "for z in xrange(1000):\n",
      "    l.insert(0, z)\n",
      "\n",
      "for y in xrange(len(l)):\n",
      "    l.pop(0)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "100 loops, best of 3: 1.52 ms per loop\n"
       ]
      }
     ],
     "prompt_number": 41
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%timeit\n",
      "# Left append - pop deque\n",
      "\n",
      "q = deque()\n",
      "for z in xrange(1000):\n",
      "   q.appendleft(z)\n",
      "\n",
      "while True:\n",
      "    try:\n",
      "        q.popleft()\n",
      "    except:\n",
      "        break"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1000 loops, best of 3: 434 us per loop\n"
       ]
      }
     ],
     "prompt_number": 42
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "a"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 43,
       "text": [
        "[9, 8, 1, 2, 3, 6]"
       ]
      }
     ],
     "prompt_number": 43
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "b = deque(a)\n",
      "b"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 44,
       "text": [
        "deque([9, 8, 1, 2, 3, 6])"
       ]
      }
     ],
     "prompt_number": 44
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "b.rotate(2)\n",
      "b"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 45,
       "text": [
        "deque([3, 6, 9, 8, 1, 2])"
       ]
      }
     ],
     "prompt_number": 45
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### `namedtuple()`\n",
      "\n",
      "A simple factory for creating subclasses of `tuple` with predefined fields which can be accesses like attributes"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from collections import namedtuple"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 46
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "Point = namedtuple(\"Point\", \"x y z\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 47
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "type(Point)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 48,
       "text": [
        "type"
       ]
      }
     ],
     "prompt_number": 48
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "p1 = Point(3,4,4)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 49
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "p1"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 50,
       "text": [
        "Point(x=3, y=4, z=4)"
       ]
      }
     ],
     "prompt_number": 50
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "Point(3,4,53,3,5)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "ename": "TypeError",
       "evalue": "__new__() takes exactly 4 arguments (6 given)",
       "output_type": "pyerr",
       "traceback": [
        "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
        "\u001b[0;32m<ipython-input-51-d2b517c9472f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mPoint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m4\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m53\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
        "\u001b[0;31mTypeError\u001b[0m: __new__() takes exactly 4 arguments (6 given)"
       ]
      }
     ],
     "prompt_number": 51
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "p1[0]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 54,
       "text": [
        "3"
       ]
      }
     ],
     "prompt_number": 54
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "p1.x"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 55,
       "text": [
        "3"
       ]
      }
     ],
     "prompt_number": 55
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The _primary_ reason to use a namedtuple instead of creating a class is to not have to type too much before one knows all the functionality of a class is needed.\n",
      "\n",
      "However, `tuple`s are much simpler than normal (unoptimized) classes, and will therefore perform better."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class ClassPoint(object):\n",
      "    def __init__(self, x, y, z):\n",
      "        self.x = x\n",
      "        self.y = y\n",
      "        self.z = z"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 56
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%timeit\n",
      "# namedtuples\n",
      "l = range(10000)\n",
      "for i in xrange(10000):\n",
      "    l[i] = Point(i, 2 * i, 3 * i)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "100 loops, best of 3: 12.5 ms per loop\n"
       ]
      }
     ],
     "prompt_number": 57
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%timeit\n",
      "# objects\n",
      "l = range(10000)\n",
      "for i in xrange(10000):\n",
      "    l[i] = ClassPoint(i, 2 * i, 3 * i)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "10 loops, best of 3: 18.1 ms per loop\n"
       ]
      }
     ],
     "prompt_number": 58
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "a = [1,2,3,4]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 60
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "a.insert(2, 0)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 61
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "aa = deque(a)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 63
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "b = deque()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 64
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "b.append(aa)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 65
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "b.append(aa)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 67
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "b"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 68,
       "text": [
        "deque([deque([1, 2, 0, 3, 4]), deque([1, 2, 0, 3, 4])])"
       ]
      }
     ],
     "prompt_number": 68
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## SciPy\n",
      "\n",
      "`pip install scipy`\n",
      "\n",
      "http://scipy.org/\n",
      "\n",
      "While NumPy provides a powerful array language interface to Python, SciPy provides functionality which uses these arrays for numerical calculations.\n",
      "\n",
      "If you're thinking of writing numerical code, it's a good idea to first check the SciPy documentation; there usually is a good implementation of what you need already.\n",
      "\n",
      "SciPy is divided in to submodules based on the kind of problem they aim to solve. Some highlights:\n",
      "\n",
      "- `scipy.linalg`\n",
      "- `scipy.sparse`\n",
      "- `scipy.integrate`\n",
      "- `scipy.optimize`\n",
      "- `scipy.interpolate`\n",
      "- `scipy.fftpack`\n",
      "- `scipy.signal`\n",
      "- `scipy.stats`\n",
      "\n",
      "The SciPy cookbook has plenty of nice examples on how to use the different components.\n",
      "\n",
      "http://scipy.org/Cookbook"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from scipy import interpolate"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 70
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "help(interpolate.PiecewisePolynomial)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Help on class PiecewisePolynomial in module scipy.interpolate.polyint:\n",
        "\n",
        "class PiecewisePolynomial(__builtin__.object)\n",
        " |  Piecewise polynomial curve specified by points and derivatives\n",
        " |  \n",
        " |  This class represents a curve that is a piecewise polynomial. It\n",
        " |  passes through a list of points and has specified derivatives at\n",
        " |  each point. The degree of the polynomial may very from segment to\n",
        " |  segment, as may the number of derivatives available. The degree\n",
        " |  should not exceed about thirty.\n",
        " |  \n",
        " |  Appending points to the end of the curve is efficient.\n",
        " |  \n",
        " |  Methods defined here:\n",
        " |  \n",
        " |  __call__(self, x)\n",
        " |      Evaluate the piecewise polynomial\n",
        " |      \n",
        " |      Parameters\n",
        " |      ----------\n",
        " |      x : scalar or array-like of length N\n",
        " |      \n",
        " |      Returns\n",
        " |      -------\n",
        " |      y : scalar or array-like of length R or length N or N by R\n",
        " |  \n",
        " |  __init__(self, xi, yi, orders=None, direction=None)\n",
        " |      Construct a piecewise polynomial\n",
        " |      \n",
        " |      Parameters\n",
        " |      ----------\n",
        " |      xi : array-like of length N\n",
        " |          a sorted list of x-coordinates\n",
        " |      yi : list of lists of length N\n",
        " |          yi[i] is the list of derivatives known at xi[i]\n",
        " |      orders : list of integers, or integer\n",
        " |          a list of polynomial orders, or a single universal order\n",
        " |      direction : {None, 1, -1}\n",
        " |          indicates whether the xi are increasing or decreasing\n",
        " |          +1 indicates increasing\n",
        " |          -1 indicates decreasing\n",
        " |          None indicates that it should be deduced from the first two xi\n",
        " |      \n",
        " |      Notes\n",
        " |      -----\n",
        " |      If orders is None, or orders[i] is None, then the degree of the\n",
        " |      polynomial segment is exactly the degree required to match all i\n",
        " |      available derivatives at both endpoints. If orders[i] is not None,\n",
        " |      then some derivatives will be ignored. The code will try to use an\n",
        " |      equal number of derivatives from each end; if the total number of\n",
        " |      derivatives needed is odd, it will prefer the rightmost endpoint. If\n",
        " |      not enough derivatives are available, an exception is raised.\n",
        " |  \n",
        " |  append(self, xi, yi, order=None)\n",
        " |      Append a single point with derivatives to the PiecewisePolynomial\n",
        " |      \n",
        " |      Parameters\n",
        " |      ----------\n",
        " |      xi : float\n",
        " |      \n",
        " |      yi : array_like\n",
        " |          yi is the list of derivatives known at xi\n",
        " |      \n",
        " |      order : integer or None\n",
        " |          a polynomial order, or instructions to use the highest\n",
        " |          possible order\n",
        " |  \n",
        " |  derivative(self, x, der)\n",
        " |      Evaluate a derivative of the piecewise polynomial\n",
        " |      \n",
        " |      Parameters\n",
        " |      ----------\n",
        " |      x : scalar or array_like of length N\n",
        " |      \n",
        " |      der : integer\n",
        " |          which single derivative to extract\n",
        " |      \n",
        " |      Returns\n",
        " |      -------\n",
        " |      y : scalar or array_like of length R or length N or N by R\n",
        " |      \n",
        " |      Notes\n",
        " |      -----\n",
        " |      This currently computes (using self.derivatives()) all derivatives\n",
        " |      of the curve segment containing each x but returns only one.\n",
        " |  \n",
        " |  derivatives(self, x, der)\n",
        " |      Evaluate a derivative of the piecewise polynomial\n",
        " |      \n",
        " |      Parameters\n",
        " |      ----------\n",
        " |      x : scalar or array_like of length N\n",
        " |      \n",
        " |      der : integer\n",
        " |          how many derivatives (including the function value as\n",
        " |          0th derivative) to extract\n",
        " |      \n",
        " |      Returns\n",
        " |      -------\n",
        " |      y : array_like of shape der by R or der by N or der by N by R\n",
        " |  \n",
        " |  extend(self, xi, yi, orders=None)\n",
        " |      Extend the PiecewisePolynomial by a list of points\n",
        " |      \n",
        " |      Parameters\n",
        " |      ----------\n",
        " |      xi : array_like of length N1\n",
        " |          a sorted list of x-coordinates\n",
        " |      yi : list of lists of length N1\n",
        " |          yi[i] is the list of derivatives known at xi[i]\n",
        " |      orders : list of integers, or integer\n",
        " |          a list of polynomial orders, or a single universal order\n",
        " |      direction : {None, 1, -1}\n",
        " |          indicates whether the xi are increasing or decreasing\n",
        " |          +1 indicates increasing\n",
        " |          -1 indicates decreasing\n",
        " |          None indicates that it should be deduced from the first two xi\n",
        " |  \n",
        " |  ----------------------------------------------------------------------\n",
        " |  Data descriptors defined here:\n",
        " |  \n",
        " |  __dict__\n",
        " |      dictionary for instance variables (if defined)\n",
        " |  \n",
        " |  __weakref__\n",
        " |      list of weak references to the object (if defined)\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 71
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In general, methods in SciPy can take an 'array-like' which will be converted to an `np.ndarray`, and then used in the SciPy modules."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## SciKits\n",
      "\n",
      "http://scikits.appspot.com/\n",
      "\n",
      "SciKits (SciPy Toolkits) are packages which builds upon SciPy, but are not included with the SciPy package for any of these reasons:\n",
      "\n",
      "- Package uses a license which is not compatible with the SciPy license\n",
      "- Package is too specialized to be worth distributing with SciPy (e.g. `hydroclimpy`)\n",
      "- Package is undergoing rapid development and can change in ways that might break backward compatibility\n",
      "\n",
      "Examples of scikits you hear a lot about\n",
      "\n",
      "- `statsmodels`\n",
      "- `scikit-image`\n",
      "- `scikit-learn`\r"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Pandas\n",
      "\n",
      "`pip install pandas`\n",
      "\n",
      "http://pandas.pydata.org/\n",
      "\n",
      "Most notably provides two data structures which builds upon NumPy arrays:\n",
      "\n",
      "`Series` for 1-dimensional data\n",
      "\n",
      "`DataFrame` for 2-dimensional data\n",
      "\n",
      "- tabular data\n",
      "- time series\n",
      "- panel data\n",
      "\n",
      "Combined with IPython, (and in particular IPython Notebook), Pandas and the numerical packages becomes great for exploring data."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### `Series`\n",
      "\n",
      "One dimensional labeled array, implemented as subclass of `ndarray`.\n",
      "\n",
      "While a `ndarray` can access elements and slices with numerical indexes, a `Series` is indexed by a list of labels for each element."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from pandas import Series"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 72
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "a = np.random.randn(5)\n",
      "a"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 73,
       "text": [
        "array([ 1.89454593,  0.90264556, -1.14669791,  0.46067578,  0.32107422])"
       ]
      }
     ],
     "prompt_number": 73
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "Series(a)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 74,
       "text": [
        "0    1.894546\n",
        "1    0.902646\n",
        "2   -1.146698\n",
        "3    0.460676\n",
        "4    0.321074"
       ]
      }
     ],
     "prompt_number": 74
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s = Series(a, index=['a', 'b', 'c', 'd', 'e'])\n",
      "s"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 75,
       "text": [
        "a    1.894546\n",
        "b    0.902646\n",
        "c   -1.146698\n",
        "d    0.460676\n",
        "e    0.321074"
       ]
      }
     ],
     "prompt_number": 75
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "a[1:4]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 76,
       "text": [
        "array([ 0.90264556, -1.14669791,  0.46067578])"
       ]
      }
     ],
     "prompt_number": 76
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s['b':'d']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 77,
       "text": [
        "b    0.902646\n",
        "c   -1.146698\n",
        "d    0.460676"
       ]
      }
     ],
     "prompt_number": 77
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In many ways a `Series` can be seen as a cross between an `ndarray` and a `dict`, and a very logical way of creating a `Series` is by passing it a `dict`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "d = {'a': 0., 'b': 1., 'c': 2.}\n",
      "s1 = Series(d)\n",
      "s1"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 78,
       "text": [
        "a    0\n",
        "b    1\n",
        "c    2"
       ]
      }
     ],
     "prompt_number": 78
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s1.index"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 79,
       "text": [
        "Index([a, b, c], dtype=object)"
       ]
      }
     ],
     "prompt_number": 79
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "(Note that the `list` we passed have been converted to an `Index` object.)\n",
      "\n",
      "In most ways a `Series` can be used just as an `ndarray`, however, when doing operations with different series, the data will automatically be aligned by the index labels"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# ndarray\n",
      "a[1:] + a[:-1]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 80,
       "text": [
        "array([ 2.79719149, -0.24405236, -0.68602213,  0.78175   ])"
       ]
      }
     ],
     "prompt_number": 80
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Series\n",
      "s[1:] + s[:-1]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 81,
       "text": [
        "a         NaN\n",
        "b    1.805291\n",
        "c   -2.293396\n",
        "d    0.921352\n",
        "e         NaN"
       ]
      }
     ],
     "prompt_number": 81
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "To help keep track of `Series`, they can be given names."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s = Series(a, index=['a', 'b', 'c', 'd', 'e'], name=\"Random example data\")\n",
      "s"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 82,
       "text": [
        "a    1.894546\n",
        "b    0.902646\n",
        "c   -1.146698\n",
        "d    0.460676\n",
        "e    0.321074\n",
        "Name: Random example data"
       ]
      }
     ],
     "prompt_number": 82
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### `DataFrame`\n",
      "\n",
      "Two-dimensional labeled data structure, most intuitively thought of as a **table**."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from pandas import DataFrame"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 83
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "A `DataFrame` can be instantiated in many ways, but the most intuitive is when constructed from a `dict` of `Series`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "d = {'one' : Series([1., 2., 3.], index=['a', 'b', 'c']), \\\n",
      "     'two' : Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])}\n",
      "\n",
      "df = DataFrame(d)\n",
      "df"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>one</th>\n",
        "      <th>two</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <td><strong>a</strong></td>\n",
        "      <td>  1</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>b</strong></td>\n",
        "      <td>  2</td>\n",
        "      <td> 2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>c</strong></td>\n",
        "      <td>  3</td>\n",
        "      <td> 3</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>d</strong></td>\n",
        "      <td>NaN</td>\n",
        "      <td> 4</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "output_type": "pyout",
       "prompt_number": 84,
       "text": [
        "   one  two\n",
        "a    1    1\n",
        "b    2    2\n",
        "c    3    3\n",
        "d  NaN    4"
       ]
      }
     ],
     "prompt_number": 84
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.index"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 85,
       "text": [
        "Index([a, b, c, d], dtype=object)"
       ]
      }
     ],
     "prompt_number": 85
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.columns"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 86,
       "text": [
        "Index([one, two], dtype=object)"
       ]
      }
     ],
     "prompt_number": 86
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df1 = DataFrame(randn(10, 4), columns=['A', 'B', 'C', 'D'])\n",
      "df2 = DataFrame(randn(7, 3), columns=['A', 'B', 'C'])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 87
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Accessing elements directly from a `DataFrame` will give you a `Series`"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df1[\"A\"]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 88,
       "text": [
        "0   -0.769256\n",
        "1   -0.392941\n",
        "2   -0.036154\n",
        "3   -0.042374\n",
        "4    1.058781\n",
        "5   -0.982873\n",
        "6    0.045609\n",
        "7   -1.608038\n",
        "8    0.670419\n",
        "9    1.020733\n",
        "Name: A"
       ]
      }
     ],
     "prompt_number": 88
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Slicing directly however only works on rows"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df1[2:4]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>A</th>\n",
        "      <th>B</th>\n",
        "      <th>C</th>\n",
        "      <th>D</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <td><strong>2</strong></td>\n",
        "      <td>-0.036154</td>\n",
        "      <td>-0.937833</td>\n",
        "      <td> 0.776257</td>\n",
        "      <td> 0.875263</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>3</strong></td>\n",
        "      <td>-0.042374</td>\n",
        "      <td>-0.715209</td>\n",
        "      <td>-0.471679</td>\n",
        "      <td>-0.189201</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "output_type": "pyout",
       "prompt_number": 89,
       "text": [
        "          A         B         C         D\n",
        "2 -0.036154 -0.937833  0.776257  0.875263\n",
        "3 -0.042374 -0.715209 -0.471679 -0.189201"
       ]
      }
     ],
     "prompt_number": 89
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "To slice in any direction, use the `ix` attribute of a `DataFrame`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df1.ix[2:,\"B\":\"D\"]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>B</th>\n",
        "      <th>C</th>\n",
        "      <th>D</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <td><strong>2</strong></td>\n",
        "      <td>-0.937833</td>\n",
        "      <td> 0.776257</td>\n",
        "      <td> 0.875263</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>3</strong></td>\n",
        "      <td>-0.715209</td>\n",
        "      <td>-0.471679</td>\n",
        "      <td>-0.189201</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>4</strong></td>\n",
        "      <td>-0.954273</td>\n",
        "      <td>-0.575008</td>\n",
        "      <td>-0.881008</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>5</strong></td>\n",
        "      <td> 0.314320</td>\n",
        "      <td> 0.351847</td>\n",
        "      <td> 0.034518</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>6</strong></td>\n",
        "      <td>-0.758584</td>\n",
        "      <td> 1.170517</td>\n",
        "      <td> 0.234722</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>7</strong></td>\n",
        "      <td> 0.096029</td>\n",
        "      <td> 0.974100</td>\n",
        "      <td>-0.860451</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>8</strong></td>\n",
        "      <td> 0.173108</td>\n",
        "      <td>-0.254731</td>\n",
        "      <td> 0.968456</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>9</strong></td>\n",
        "      <td> 0.030799</td>\n",
        "      <td> 0.231640</td>\n",
        "      <td> 0.212490</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "output_type": "pyout",
       "prompt_number": 90,
       "text": [
        "          B         C         D\n",
        "2 -0.937833  0.776257  0.875263\n",
        "3 -0.715209 -0.471679 -0.189201\n",
        "4 -0.954273 -0.575008 -0.881008\n",
        "5  0.314320  0.351847  0.034518\n",
        "6 -0.758584  1.170517  0.234722\n",
        "7  0.096029  0.974100 -0.860451\n",
        "8  0.173108 -0.254731  0.968456\n",
        "9  0.030799  0.231640  0.212490"
       ]
      }
     ],
     "prompt_number": 90
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "To get a **row** as series, use the `xs` method"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df1.xs(5)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 91,
       "text": [
        "A   -0.982873\n",
        "B    0.314320\n",
        "C    0.351847\n",
        "D    0.034518\n",
        "Name: 5"
       ]
      }
     ],
     "prompt_number": 91
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df1 + df2"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>A</th>\n",
        "      <th>B</th>\n",
        "      <th>C</th>\n",
        "      <th>D</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <td><strong>0</strong></td>\n",
        "      <td>-1.390004</td>\n",
        "      <td> 1.357910</td>\n",
        "      <td>-0.212756</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>1</strong></td>\n",
        "      <td> 0.579743</td>\n",
        "      <td>-2.053898</td>\n",
        "      <td> 0.110167</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>2</strong></td>\n",
        "      <td> 0.291970</td>\n",
        "      <td>-1.828201</td>\n",
        "      <td> 1.277833</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>3</strong></td>\n",
        "      <td> 0.151515</td>\n",
        "      <td> 0.590763</td>\n",
        "      <td>-2.604155</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>4</strong></td>\n",
        "      <td> 0.466383</td>\n",
        "      <td>-0.067969</td>\n",
        "      <td>-0.152371</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>5</strong></td>\n",
        "      <td>-1.005964</td>\n",
        "      <td> 0.394166</td>\n",
        "      <td>-0.380423</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>6</strong></td>\n",
        "      <td>-1.342940</td>\n",
        "      <td>-1.582353</td>\n",
        "      <td> 1.359328</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>7</strong></td>\n",
        "      <td>      NaN</td>\n",
        "      <td>      NaN</td>\n",
        "      <td>      NaN</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>8</strong></td>\n",
        "      <td>      NaN</td>\n",
        "      <td>      NaN</td>\n",
        "      <td>      NaN</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>9</strong></td>\n",
        "      <td>      NaN</td>\n",
        "      <td>      NaN</td>\n",
        "      <td>      NaN</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "output_type": "pyout",
       "prompt_number": 92,
       "text": [
        "          A         B         C   D\n",
        "0 -1.390004  1.357910 -0.212756 NaN\n",
        "1  0.579743 -2.053898  0.110167 NaN\n",
        "2  0.291970 -1.828201  1.277833 NaN\n",
        "3  0.151515  0.590763 -2.604155 NaN\n",
        "4  0.466383 -0.067969 -0.152371 NaN\n",
        "5 -1.005964  0.394166 -0.380423 NaN\n",
        "6 -1.342940 -1.582353  1.359328 NaN\n",
        "7       NaN       NaN       NaN NaN\n",
        "8       NaN       NaN       NaN NaN\n",
        "9       NaN       NaN       NaN NaN"
       ]
      }
     ],
     "prompt_number": 92
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df1 * 20"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>A</th>\n",
        "      <th>B</th>\n",
        "      <th>C</th>\n",
        "      <th>D</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <td><strong>0</strong></td>\n",
        "      <td>-15.385119</td>\n",
        "      <td>  2.827629</td>\n",
        "      <td>-14.666131</td>\n",
        "      <td>-24.950579</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>1</strong></td>\n",
        "      <td> -7.858822</td>\n",
        "      <td>-21.900526</td>\n",
        "      <td>  7.637977</td>\n",
        "      <td>  4.545096</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>2</strong></td>\n",
        "      <td> -0.723078</td>\n",
        "      <td>-18.756662</td>\n",
        "      <td> 15.525133</td>\n",
        "      <td> 17.505269</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>3</strong></td>\n",
        "      <td> -0.847483</td>\n",
        "      <td>-14.304187</td>\n",
        "      <td> -9.433586</td>\n",
        "      <td> -3.784014</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>4</strong></td>\n",
        "      <td> 21.175615</td>\n",
        "      <td>-19.085451</td>\n",
        "      <td>-11.500155</td>\n",
        "      <td>-17.620163</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>5</strong></td>\n",
        "      <td>-19.657465</td>\n",
        "      <td>  6.286394</td>\n",
        "      <td>  7.036946</td>\n",
        "      <td>  0.690362</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>6</strong></td>\n",
        "      <td>  0.912172</td>\n",
        "      <td>-15.171680</td>\n",
        "      <td> 23.410348</td>\n",
        "      <td>  4.694435</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>7</strong></td>\n",
        "      <td>-32.160761</td>\n",
        "      <td>  1.920575</td>\n",
        "      <td> 19.481998</td>\n",
        "      <td>-17.209029</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>8</strong></td>\n",
        "      <td> 13.408376</td>\n",
        "      <td>  3.462153</td>\n",
        "      <td> -5.094613</td>\n",
        "      <td> 19.369123</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>9</strong></td>\n",
        "      <td> 20.414662</td>\n",
        "      <td>  0.615983</td>\n",
        "      <td>  4.632805</td>\n",
        "      <td>  4.249790</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "output_type": "pyout",
       "prompt_number": 93,
       "text": [
        "           A          B          C          D\n",
        "0 -15.385119   2.827629 -14.666131 -24.950579\n",
        "1  -7.858822 -21.900526   7.637977   4.545096\n",
        "2  -0.723078 -18.756662  15.525133  17.505269\n",
        "3  -0.847483 -14.304187  -9.433586  -3.784014\n",
        "4  21.175615 -19.085451 -11.500155 -17.620163\n",
        "5 -19.657465   6.286394   7.036946   0.690362\n",
        "6   0.912172 -15.171680  23.410348   4.694435\n",
        "7 -32.160761   1.920575  19.481998 -17.209029\n",
        "8  13.408376   3.462153  -5.094613  19.369123\n",
        "9  20.414662   0.615983   4.632805   4.249790"
       ]
      }
     ],
     "prompt_number": 93
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Broadcasting with Series is done row wise:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df1 + df2.xs(2)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>A</th>\n",
        "      <th>B</th>\n",
        "      <th>C</th>\n",
        "      <th>D</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <td><strong>0</strong></td>\n",
        "      <td>-0.441132</td>\n",
        "      <td>-0.748987</td>\n",
        "      <td>-0.231730</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>1</strong></td>\n",
        "      <td>-0.064817</td>\n",
        "      <td>-1.985395</td>\n",
        "      <td> 0.883475</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>2</strong></td>\n",
        "      <td> 0.291970</td>\n",
        "      <td>-1.828201</td>\n",
        "      <td> 1.277833</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>3</strong></td>\n",
        "      <td> 0.285750</td>\n",
        "      <td>-1.605578</td>\n",
        "      <td> 0.029897</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>4</strong></td>\n",
        "      <td> 1.386905</td>\n",
        "      <td>-1.844641</td>\n",
        "      <td>-0.073431</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>5</strong></td>\n",
        "      <td>-0.654749</td>\n",
        "      <td>-0.576049</td>\n",
        "      <td> 0.853424</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>6</strong></td>\n",
        "      <td> 0.373733</td>\n",
        "      <td>-1.648952</td>\n",
        "      <td> 1.672094</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>7</strong></td>\n",
        "      <td>-1.279914</td>\n",
        "      <td>-0.794340</td>\n",
        "      <td> 1.475676</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>8</strong></td>\n",
        "      <td> 0.998543</td>\n",
        "      <td>-0.717261</td>\n",
        "      <td> 0.246846</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>9</strong></td>\n",
        "      <td> 1.348857</td>\n",
        "      <td>-0.859569</td>\n",
        "      <td> 0.733217</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "output_type": "pyout",
       "prompt_number": 94,
       "text": [
        "          A         B         C   D\n",
        "0 -0.441132 -0.748987 -0.231730 NaN\n",
        "1 -0.064817 -1.985395  0.883475 NaN\n",
        "2  0.291970 -1.828201  1.277833 NaN\n",
        "3  0.285750 -1.605578  0.029897 NaN\n",
        "4  1.386905 -1.844641 -0.073431 NaN\n",
        "5 -0.654749 -0.576049  0.853424 NaN\n",
        "6  0.373733 -1.648952  1.672094 NaN\n",
        "7 -1.279914 -0.794340  1.475676 NaN\n",
        "8  0.998543 -0.717261  0.246846 NaN\n",
        "9  1.348857 -0.859569  0.733217 NaN"
       ]
      }
     ],
     "prompt_number": 94
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "For a `DataFrame`, each **column** must have the same type (unlike a 2d `ndarray`).\n",
      "\n",
      "When all columns have numerical values, NumPy functions can be used on the `DataFrame` just as if it was a 2d `ndarray`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "np.exp(df1)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>A</th>\n",
        "      <th>B</th>\n",
        "      <th>C</th>\n",
        "      <th>D</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <td><strong>0</strong></td>\n",
        "      <td> 0.463358</td>\n",
        "      <td> 1.151864</td>\n",
        "      <td> 0.480318</td>\n",
        "      <td> 0.287214</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>1</strong></td>\n",
        "      <td> 0.675069</td>\n",
        "      <td> 0.334531</td>\n",
        "      <td> 1.465064</td>\n",
        "      <td> 1.255150</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>2</strong></td>\n",
        "      <td> 0.964492</td>\n",
        "      <td> 0.391475</td>\n",
        "      <td> 2.173322</td>\n",
        "      <td> 2.399507</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>3</strong></td>\n",
        "      <td> 0.958511</td>\n",
        "      <td> 0.489090</td>\n",
        "      <td> 0.623954</td>\n",
        "      <td> 0.827620</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>4</strong></td>\n",
        "      <td> 2.882854</td>\n",
        "      <td> 0.385092</td>\n",
        "      <td> 0.562701</td>\n",
        "      <td> 0.414365</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>5</strong></td>\n",
        "      <td> 0.374234</td>\n",
        "      <td> 1.369327</td>\n",
        "      <td> 1.421691</td>\n",
        "      <td> 1.035121</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>6</strong></td>\n",
        "      <td> 1.046665</td>\n",
        "      <td> 0.468329</td>\n",
        "      <td> 3.223660</td>\n",
        "      <td> 1.264557</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>7</strong></td>\n",
        "      <td> 0.200280</td>\n",
        "      <td> 1.100791</td>\n",
        "      <td> 2.648782</td>\n",
        "      <td> 0.422971</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>8</strong></td>\n",
        "      <td> 1.955056</td>\n",
        "      <td> 1.188994</td>\n",
        "      <td> 0.775125</td>\n",
        "      <td> 2.633875</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>9</strong></td>\n",
        "      <td> 2.775229</td>\n",
        "      <td> 1.031278</td>\n",
        "      <td> 1.260666</td>\n",
        "      <td> 1.236753</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "output_type": "pyout",
       "prompt_number": 95,
       "text": [
        "          A         B         C         D\n",
        "0  0.463358  1.151864  0.480318  0.287214\n",
        "1  0.675069  0.334531  1.465064  1.255150\n",
        "2  0.964492  0.391475  2.173322  2.399507\n",
        "3  0.958511  0.489090  0.623954  0.827620\n",
        "4  2.882854  0.385092  0.562701  0.414365\n",
        "5  0.374234  1.369327  1.421691  1.035121\n",
        "6  1.046665  0.468329  3.223660  1.264557\n",
        "7  0.200280  1.100791  2.648782  0.422971\n",
        "8  1.955056  1.188994  0.775125  2.633875\n",
        "9  2.775229  1.031278  1.260666  1.236753"
       ]
      }
     ],
     "prompt_number": 95
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This means `DataFrames` works almost seemlessly with SciPy or SciKits. In particular developers of statistics SciKits have started putting a lot of effort in to making their packages as Pandas compatible as possible."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "It should be noted that Pandas also has a `Panel` data structure, which is a 3-dimensional structure which can be viewed conceptually as a \"Series of DataFrames\".\n",
      "\n",
      "In addition to this there are experimental data structures `Panel4D` and `PanelND`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Assignment for Session 4\n",
      "\n",
      "### Github API\n",
      "\n",
      "Github has a very extensive RESTful API (a web service). For example, to get information about members of an organization, one can do this"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import requests"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 96
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "with open(\"secret\") as secret:\n",
      "    password = secret.read().strip()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "ename": "IOError",
       "evalue": "[Errno 2] No such file or directory: 'secret'",
       "output_type": "pyerr",
       "traceback": [
        "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mIOError\u001b[0m                                   Traceback (most recent call last)",
        "\u001b[0;32m<ipython-input-97-ce01dca0116a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"secret\"\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0msecret\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m     \u001b[0mpassword\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msecret\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstrip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
        "\u001b[0;31mIOError\u001b[0m: [Errno 2] No such file or directory: 'secret'"
       ]
      }
     ],
     "prompt_number": 97
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "users = requests.get(\"https://api.github.com/orgs/pythonkurs/members\", auth=(\"vals\", password))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 98
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### NEVER EVER PUT CREDENTIALS AS TEXT IN SCRIPTS!"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "users_data = users.json()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 99
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "len(users_data)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 100,
       "text": [
        "48"
       ]
      }
     ],
     "prompt_number": 100
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "users_data[0]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 101,
       "text": [
        "{u'avatar_url': u'https://secure.gravatar.com/avatar/e2a0accefc4b4298a35e76da78a0f52c?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-user-420.png',\n",
        " u'events_url': u'https://api.github.com/users/alneberg/events{/privacy}',\n",
        " u'followers_url': u'https://api.github.com/users/alneberg/followers',\n",
        " u'following_url': u'https://api.github.com/users/alneberg/following',\n",
        " u'gists_url': u'https://api.github.com/users/alneberg/gists{/gist_id}',\n",
        " u'gravatar_id': u'e2a0accefc4b4298a35e76da78a0f52c',\n",
        " u'id': 1250075,\n",
        " u'login': u'alneberg',\n",
        " u'organizations_url': u'https://api.github.com/users/alneberg/orgs',\n",
        " u'received_events_url': u'https://api.github.com/users/alneberg/received_events',\n",
        " u'repos_url': u'https://api.github.com/users/alneberg/repos',\n",
        " u'starred_url': u'https://api.github.com/users/alneberg/starred{/owner}{/repo}',\n",
        " u'subscriptions_url': u'https://api.github.com/users/alneberg/subscriptions',\n",
        " u'type': u'User',\n",
        " u'url': u'https://api.github.com/users/alneberg'}"
       ]
      }
     ],
     "prompt_number": 101
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The API is documented very clearly and with lots of examples at http://developer.github.com/\n",
      "\n",
      "Note in particular that URI's to related resources are given with keys of the form `*_url`\n",
      "\n",
      "There are (at the moment of writing) 28 repositories in the pythonkurs organization.\n",
      "\n",
      "### 1.\n",
      "\n",
      "Use the Github API to get the commit history of each repository in the organization in to a DataFrame, where columns are repositories and rows are commits indexed by time. The content should be the commit message.\n",
      "\n",
      "Depending on strategy for getting the commit times, you might want to use the third party package `dateutils` to parse the \"date\" values. It is installable by \n",
      "\n",
      "    pip install dateutil"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from dateutil import parser"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 102
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "parser.parse(\"2013-02-04T16:58:05Z\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 103,
       "text": [
        "datetime.datetime(2013, 2, 4, 16, 58, 5, tzinfo=tzutc())"
       ]
      }
     ],
     "prompt_number": 103
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "For clarification, this could be how one `Series` in the `DataFrame` would look like"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import datetime\n",
      "\n",
      "base = datetime.datetime.now()\n",
      "date_list = [base - datetime.timedelta(days=x) for x in range(5)]\n",
      "\n",
      "s = Series([\"A commit message\"] * 5, index=date_list, name=\"A repo\")\n",
      "s"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 104,
       "text": [
        "2013-02-06 15:39:58.265767    A commit message\n",
        "2013-02-05 15:39:58.265767    A commit message\n",
        "2013-02-04 15:39:58.265767    A commit message\n",
        "2013-02-03 15:39:58.265767    A commit message\n",
        "2013-02-02 15:39:58.265767    A commit message\n",
        "Name: A repo"
       ]
      }
     ],
     "prompt_number": 104
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "DataFrame(s)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>A repo</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <td><strong>2013-02-06 15:39:58.265767</strong></td>\n",
        "      <td> A commit message</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>2013-02-05 15:39:58.265767</strong></td>\n",
        "      <td> A commit message</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>2013-02-04 15:39:58.265767</strong></td>\n",
        "      <td> A commit message</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>2013-02-03 15:39:58.265767</strong></td>\n",
        "      <td> A commit message</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><strong>2013-02-02 15:39:58.265767</strong></td>\n",
        "      <td> A commit message</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "output_type": "pyout",
       "prompt_number": 105,
       "text": [
        "                                      A repo\n",
        "2013-02-06 15:39:58.265767  A commit message\n",
        "2013-02-05 15:39:58.265767  A commit message\n",
        "2013-02-04 15:39:58.265767  A commit message\n",
        "2013-02-03 15:39:58.265767  A commit message\n",
        "2013-02-02 15:39:58.265767  A commit message"
       ]
      }
     ],
     "prompt_number": 105
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Put a function which returns the specified `DataFrame` in a submodule.\n",
      "\n",
      "NOTE: DO NOT PUT ANY CREDENTIALS IN ANY FILES ADDED TO THE REPO. You need to figure out a good way of handling the authentication. (The _best_ way is to use OAuth, but it is rather complicated.) One solution could for example be to make the function take credentials as an argument."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### 2.\n",
      "\n",
      "Use the `DataFrame` to figure out **what is the most common weekday and hour of a day to commit to a course repo**.\n",
      "\n",
      "To do this, you might want to do some interactive exploring of the data. This can be done in the standard Python interpreter, but your life might become a lot easier if you use IPython. IPython can be installed by\n",
      "    \n",
      "    pip install ipython\n",
      "\n",
      "and can then be started by running `ipython` in stead of `python`.\n",
      "\n",
      "You might also want to try the IPython Notebook. In that case you also need to install a couple of dependencies by doing\n",
      "\n",
      "    pip install tornado\n",
      "    pip install pyzmq\n",
      "\n",
      "You _should_ then be able to start the notebook by running\n",
      "\n",
      "    ipython notebook --pylab=inline\n",
      "\n",
      "If it doesn't work, don't worry about it, just use the normal IPython.\n",
      "\n",
      "When you have figured out a way of giving a value for the most common weekday and hour of a day, write a function in a `session4` submodule which takes a `DataFrame` as argument and returns the weekday and hour of a day."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### 3.\n",
      "\n",
      "When you have learned how to get data from the Github API, spend some time playing around with data you can get and try to figure out something else about the pythonkurs organization. \n",
      "\n",
      "No need to write functions or so for this assignment, just explore. Then write a _very_ short plain text file in the root of your repository called **`pythonkurs_organization.txt`** with some information about the data that you found interesting. \n",
      "\n",
      "By \"_very_ short\", we mean **no longer than 150 words**. Putting more than 150 words in the text file will count as failing the assignment.\n",
      "\n",
      "Of course, \"interesting\" is subjective. The important part is showing that you explored the data some."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "After this assignment, your repository should contain these files:\n",
      "\n",
      "    surname/\n",
      "        surname/\n",
      "            __init__.py\n",
      "            session2.py\n",
      "            session3.py\n",
      "            session4.py    <- new\n",
      "\n",
      "        scripts/\n",
      "            getting_data.py\n",
      "            check_repo.py\n",
      "        \n",
      "        README.md\n",
      "        setup.py\n",
      "        pythonkurs_organization.txt    <- new\n",
      "\n",
      "(As well as other files, but these are the ones we have talked about.)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df1.shape"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 108,
       "text": [
        "(10, 4)"
       ]
      }
     ],
     "prompt_number": 108
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.ix[:7, :3].shape"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 109,
       "text": [
        "(4, 2)"
       ]
      }
     ],
     "prompt_number": 109
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}