11.Name Gender Identifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Name Gender Identifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Building a feature extractor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An idea is to use the last letter of the name to predict the gender. For instance, names ending in *a*, *e* and *i* are likely to be female, while names ending in *k*, *o*, *r*, *s* and *t* are likely to be male."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'last_letter': 'n'}"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Feature extractor\n",
    "def gender_features(word):\n",
    "    return {'last_letter': word[-1]}\n",
    "\n",
    "gender_features('John')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The returned dictionary is known as a **feature set**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Exploring the `names` corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Names Corpus, Version 1.3 (1994-03-29) Copyright (C) 1991 Mark Kantrowitz Additions by Bill Ross  This corpus contains 5001 female names and 2943 male names, sorted alphabetically, one per line.  You may use the lists of names for any purpose, so long as credit is given in any published work. You may also redistribute the list if you provide the recipients with a copy of this README file. The lists are not in the public domain (I retain the copyright on the lists) but are freely redistributable.  If you have any additions to the lists of names, I would appreciate receiving them.  Mark Kantrowitz <mkant+@cs.cmu.edu> http://www-2.cs.cmu.edu/afs/cs/project/ai-repository/ai/areas/nlp/corpora/names/'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import names\n",
    "\n",
    "names.readme().replace('\\n', ' ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['female.txt', 'male.txt']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "names.fileids()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Abagael', 'Abagail', 'Abbe', 'Abbey', 'Abbi']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "names.words('female.txt')[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Building the classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to prepare a list of examples and corresponding class labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Abagael', 'female'),\n",
       " ('Abagail', 'female'),\n",
       " ('Abbe', 'female'),\n",
       " ('Abbey', 'female'),\n",
       " ('Abbi', 'female')]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labeled_names = ([(name, 'female') for name in names.words('female.txt')] + [(name, 'male') for name in names.words('male.txt')])\n",
    "labeled_names[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Lilli', 'female'),\n",
       " ('Jeff', 'male'),\n",
       " ('Kaspar', 'male'),\n",
       " ('Celia', 'female'),\n",
       " ('Renell', 'female')]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import random\n",
    "random.shuffle(labeled_names) # We shuffle the data so that we can split it by index into training and test data.\n",
    "labeled_names[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[({'last_letter': 'i'}, 'female'),\n",
       " ({'last_letter': 'f'}, 'male'),\n",
       " ({'last_letter': 'r'}, 'male'),\n",
       " ({'last_letter': 'a'}, 'female'),\n",
       " ({'last_letter': 'l'}, 'female')]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "featuresets = [(gender_features(n), gender) for (n, gender) in labeled_names]\n",
    "featuresets[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7944"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(featuresets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk import NaiveBayesClassifier\n",
    "\n",
    "# We split the data into a training (80%) and test (20%) set:\n",
    "TRAIN_SET_SIZE = round(len(featuresets) * .8)\n",
    "train_set, test_set = featuresets[:TRAIN_SET_SIZE], featuresets[TRAIN_SET_SIZE:]\n",
    "\n",
    "# We also get the names in the test set, to be used later:\n",
    "test_names = labeled_names[TRAIN_SET_SIZE:]\n",
    "\n",
    "classifier = NaiveBayesClassifier.train(train_set)\n",
    "\n",
    "# When working with large corpora, constructing a single list that contains the features of every instance can use up a large amount of memory. In these cases, use the function nltk.classify.apply_features, which returns an object that acts like a list but does not store all the feature sets in memory: \n",
    "# from nltk.classify import apply_features\n",
    "# train_names, test_names = labeled_names[:round(len(featuresets) * .8)], labeled_names[round(len(featuresets) * .8):]\n",
    "# train_set = apply_features(gender_features, labeled_names[500:])\n",
    "# test_set = apply_features(gender_features, labeled_names[:500])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Most Informative Features\n",
      "             last_letter = 'a'            female : male   =     34.1 : 1.0\n",
      "             last_letter = 'k'              male : female =     28.0 : 1.0\n",
      "             last_letter = 'v'              male : female =     16.5 : 1.0\n",
      "             last_letter = 'f'              male : female =     12.6 : 1.0\n",
      "             last_letter = 'm'              male : female =     10.7 : 1.0\n",
      "             last_letter = 'p'              male : female =      9.2 : 1.0\n",
      "             last_letter = 'd'              male : female =      8.9 : 1.0\n",
      "             last_letter = 'o'              male : female =      8.6 : 1.0\n",
      "             last_letter = 'r'              male : female =      7.2 : 1.0\n",
      "             last_letter = 'g'              male : female =      5.9 : 1.0\n"
     ]
    }
   ],
   "source": [
    "classifier.show_most_informative_features(10) # Prints likelihood ratios for most informative features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Testing the classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['female', 'male']"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier.labels()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.76"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.classify import accuracy\n",
    "\n",
    "round(accuracy(classifier, test_set), 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'female'"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier.classify(gender_features('Aphrodite'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'male'"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier.classify(gender_features('Zeus'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Building a classifier with more features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'count(a)': 0,\n",
       " 'count(b)': 0,\n",
       " 'count(c)': 0,\n",
       " 'count(d)': 0,\n",
       " 'count(e)': 0,\n",
       " 'count(f)': 0,\n",
       " 'count(g)': 0,\n",
       " 'count(h)': 1,\n",
       " 'count(i)': 0,\n",
       " 'count(j)': 1,\n",
       " 'count(k)': 0,\n",
       " 'count(l)': 0,\n",
       " 'count(m)': 0,\n",
       " 'count(n)': 1,\n",
       " 'count(o)': 1,\n",
       " 'count(p)': 0,\n",
       " 'count(q)': 0,\n",
       " 'count(r)': 0,\n",
       " 'count(s)': 0,\n",
       " 'count(t)': 0,\n",
       " 'count(u)': 0,\n",
       " 'count(v)': 0,\n",
       " 'count(w)': 0,\n",
       " 'count(x)': 0,\n",
       " 'count(y)': 0,\n",
       " 'count(z)': 0,\n",
       " 'first_letter': 'j',\n",
       " 'has(a)': False,\n",
       " 'has(b)': False,\n",
       " 'has(c)': False,\n",
       " 'has(d)': False,\n",
       " 'has(e)': False,\n",
       " 'has(f)': False,\n",
       " 'has(g)': False,\n",
       " 'has(h)': True,\n",
       " 'has(i)': False,\n",
       " 'has(j)': True,\n",
       " 'has(k)': False,\n",
       " 'has(l)': False,\n",
       " 'has(m)': False,\n",
       " 'has(n)': True,\n",
       " 'has(o)': True,\n",
       " 'has(p)': False,\n",
       " 'has(q)': False,\n",
       " 'has(r)': False,\n",
       " 'has(s)': False,\n",
       " 'has(t)': False,\n",
       " 'has(u)': False,\n",
       " 'has(v)': False,\n",
       " 'has(w)': False,\n",
       " 'has(x)': False,\n",
       " 'has(y)': False,\n",
       " 'has(z)': False,\n",
       " 'last_letter': 'n'}"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def gender_features2(name):\n",
    "    features = {}\n",
    "    features[\"first_letter\"] = name[0].lower()\n",
    "    features[\"last_letter\"] = name[-1].lower()\n",
    "    for letter in 'abcdefghijklmnopqrstuvwxyz':\n",
    "        features[\"count({})\".format(letter)] = name.lower().count(letter)\n",
    "        features[\"has({})\".format(letter)] = (letter in name.lower())\n",
    "    return features\n",
    "\n",
    "gender_features2('John')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'count(a)': 0,\n",
       "  'count(b)': 0,\n",
       "  'count(c)': 0,\n",
       "  'count(d)': 0,\n",
       "  'count(e)': 0,\n",
       "  'count(f)': 0,\n",
       "  'count(g)': 0,\n",
       "  'count(h)': 0,\n",
       "  'count(i)': 2,\n",
       "  'count(j)': 0,\n",
       "  'count(k)': 0,\n",
       "  'count(l)': 3,\n",
       "  'count(m)': 0,\n",
       "  'count(n)': 0,\n",
       "  'count(o)': 0,\n",
       "  'count(p)': 0,\n",
       "  'count(q)': 0,\n",
       "  'count(r)': 0,\n",
       "  'count(s)': 0,\n",
       "  'count(t)': 0,\n",
       "  'count(u)': 0,\n",
       "  'count(v)': 0,\n",
       "  'count(w)': 0,\n",
       "  'count(x)': 0,\n",
       "  'count(y)': 0,\n",
       "  'count(z)': 0,\n",
       "  'first_letter': 'l',\n",
       "  'has(a)': False,\n",
       "  'has(b)': False,\n",
       "  'has(c)': False,\n",
       "  'has(d)': False,\n",
       "  'has(e)': False,\n",
       "  'has(f)': False,\n",
       "  'has(g)': False,\n",
       "  'has(h)': False,\n",
       "  'has(i)': True,\n",
       "  'has(j)': False,\n",
       "  'has(k)': False,\n",
       "  'has(l)': True,\n",
       "  'has(m)': False,\n",
       "  'has(n)': False,\n",
       "  'has(o)': False,\n",
       "  'has(p)': False,\n",
       "  'has(q)': False,\n",
       "  'has(r)': False,\n",
       "  'has(s)': False,\n",
       "  'has(t)': False,\n",
       "  'has(u)': False,\n",
       "  'has(v)': False,\n",
       "  'has(w)': False,\n",
       "  'has(x)': False,\n",
       "  'has(y)': False,\n",
       "  'has(z)': False,\n",
       "  'last_letter': 'i'},\n",
       " 'female')"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "featuresets2 = [(gender_features2(n), gender) for (n, gender) in labeled_names]\n",
    "featuresets2[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.78"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set2, test_set2 = featuresets2[:TRAIN_SET_SIZE], featuresets2[TRAIN_SET_SIZE:]\n",
    "classifier2 = NaiveBayesClassifier.train(train_set2)\n",
    "round(accuracy(classifier2, test_set2), 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We would have expected that having too many specific features on a small dataset would lead to overfitting, but it seems the classifier was good at avoiding that since its performance is slightly better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Most Informative Features\n",
      "             last_letter = 'a'            female : male   =     34.1 : 1.0\n",
      "             last_letter = 'k'              male : female =     28.0 : 1.0\n",
      "             last_letter = 'v'              male : female =     16.5 : 1.0\n",
      "             last_letter = 'f'              male : female =     12.6 : 1.0\n",
      "             last_letter = 'm'              male : female =     10.7 : 1.0\n",
      "             last_letter = 'p'              male : female =      9.2 : 1.0\n",
      "             last_letter = 'd'              male : female =      8.9 : 1.0\n",
      "             last_letter = 'o'              male : female =      8.6 : 1.0\n",
      "             last_letter = 'r'              male : female =      7.2 : 1.0\n",
      "                count(v) = 2              female : male   =      6.8 : 1.0\n",
      "             last_letter = 'g'              male : female =      5.9 : 1.0\n",
      "             last_letter = 'w'              male : female =      5.9 : 1.0\n",
      "                count(a) = 3              female : male   =      5.4 : 1.0\n",
      "                count(i) = 3                male : female =      4.4 : 1.0\n",
      "            first_letter = 'w'              male : female =      4.4 : 1.0\n"
     ]
    }
   ],
   "source": [
    "classifier2.show_most_informative_features(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, it seems the classifier is mainly using the last letter, along with some other features that happen to improve the accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Comparing the two classifiers using `nltk.metrics`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we start, here's a useful function for comparing strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.metrics import edit_distance\n",
    "\n",
    "edit_distance(\"John\", \"Joan\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The NLTK metrics module provides functions for calculating metrics beyond mere accuracy. But in order to do so, we need to build 2 sets for each classification label: a reference set of correct values, and a test set of observed values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "import collections\n",
    "\n",
    "# Classifier 1\n",
    "refsets = collections.defaultdict(set) # For what this is: https://stackoverflow.com/questions/5900578/how-does-collections-defaultdict-work\n",
    "testsets = collections.defaultdict(set)\n",
    "\n",
    "for i, (feats, label) in enumerate(test_set):\n",
    "    refsets[label].add(i)\n",
    "    observed = classifier.classify(feats)\n",
    "    testsets[observed].add(i)\n",
    "    \n",
    "# Classifier 2\n",
    "refsets2 = collections.defaultdict(set)\n",
    "testsets2 = collections.defaultdict(set)\n",
    "\n",
    "for i, (feats, label) in enumerate(test_set2):\n",
    "    refsets2[label].add(i)\n",
    "    observed = classifier2.classify(feats)\n",
    "    testsets2[observed].add(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defaultdict(set,\n",
       "            {'female': {2,\n",
       "              3,\n",
       "              4,\n",
       "              6,\n",
       "              7,\n",
       "              8,\n",
       "              9,\n",
       "              10,\n",
       "              11,\n",
       "              12,\n",
       "              17,\n",
       "              19,\n",
       "              21,\n",
       "              22,\n",
       "              23,\n",
       "              24,\n",
       "              25,\n",
       "              28,\n",
       "              29,\n",
       "              32,\n",
       "              33,\n",
       "              34,\n",
       "              38,\n",
       "              40,\n",
       "              41,\n",
       "              44,\n",
       "              47,\n",
       "              48,\n",
       "              50,\n",
       "              51,\n",
       "              53,\n",
       "              56,\n",
       "              57,\n",
       "              60,\n",
       "              61,\n",
       "              62,\n",
       "              66,\n",
       "              67,\n",
       "              68,\n",
       "              69,\n",
       "              70,\n",
       "              72,\n",
       "              74,\n",
       "              75,\n",
       "              78,\n",
       "              79,\n",
       "              80,\n",
       "              82,\n",
       "              83,\n",
       "              85,\n",
       "              87,\n",
       "              88,\n",
       "              89,\n",
       "              90,\n",
       "              91,\n",
       "              92,\n",
       "              93,\n",
       "              95,\n",
       "              96,\n",
       "              98,\n",
       "              104,\n",
       "              107,\n",
       "              108,\n",
       "              109,\n",
       "              111,\n",
       "              113,\n",
       "              114,\n",
       "              115,\n",
       "              116,\n",
       "              117,\n",
       "              119,\n",
       "              120,\n",
       "              121,\n",
       "              122,\n",
       "              123,\n",
       "              124,\n",
       "              125,\n",
       "              126,\n",
       "              127,\n",
       "              128,\n",
       "              129,\n",
       "              130,\n",
       "              131,\n",
       "              132,\n",
       "              134,\n",
       "              135,\n",
       "              136,\n",
       "              137,\n",
       "              138,\n",
       "              139,\n",
       "              140,\n",
       "              141,\n",
       "              143,\n",
       "              150,\n",
       "              153,\n",
       "              154,\n",
       "              157,\n",
       "              159,\n",
       "              160,\n",
       "              163,\n",
       "              164,\n",
       "              167,\n",
       "              169,\n",
       "              171,\n",
       "              172,\n",
       "              173,\n",
       "              176,\n",
       "              177,\n",
       "              178,\n",
       "              179,\n",
       "              181,\n",
       "              182,\n",
       "              183,\n",
       "              185,\n",
       "              186,\n",
       "              187,\n",
       "              190,\n",
       "              192,\n",
       "              193,\n",
       "              194,\n",
       "              197,\n",
       "              198,\n",
       "              199,\n",
       "              200,\n",
       "              201,\n",
       "              202,\n",
       "              204,\n",
       "              205,\n",
       "              208,\n",
       "              210,\n",
       "              211,\n",
       "              212,\n",
       "              213,\n",
       "              214,\n",
       "              215,\n",
       "              216,\n",
       "              217,\n",
       "              218,\n",
       "              219,\n",
       "              221,\n",
       "              222,\n",
       "              224,\n",
       "              225,\n",
       "              228,\n",
       "              230,\n",
       "              231,\n",
       "              232,\n",
       "              234,\n",
       "              235,\n",
       "              236,\n",
       "              237,\n",
       "              240,\n",
       "              241,\n",
       "              242,\n",
       "              243,\n",
       "              244,\n",
       "              245,\n",
       "              246,\n",
       "              247,\n",
       "              248,\n",
       "              249,\n",
       "              250,\n",
       "              255,\n",
       "              257,\n",
       "              258,\n",
       "              260,\n",
       "              262,\n",
       "              264,\n",
       "              270,\n",
       "              274,\n",
       "              277,\n",
       "              278,\n",
       "              279,\n",
       "              280,\n",
       "              281,\n",
       "              282,\n",
       "              283,\n",
       "              285,\n",
       "              287,\n",
       "              288,\n",
       "              289,\n",
       "              290,\n",
       "              291,\n",
       "              292,\n",
       "              293,\n",
       "              295,\n",
       "              298,\n",
       "              299,\n",
       "              302,\n",
       "              303,\n",
       "              304,\n",
       "              306,\n",
       "              309,\n",
       "              310,\n",
       "              312,\n",
       "              313,\n",
       "              314,\n",
       "              317,\n",
       "              319,\n",
       "              320,\n",
       "              321,\n",
       "              322,\n",
       "              324,\n",
       "              325,\n",
       "              326,\n",
       "              329,\n",
       "              333,\n",
       "              334,\n",
       "              337,\n",
       "              339,\n",
       "              340,\n",
       "              341,\n",
       "              342,\n",
       "              343,\n",
       "              345,\n",
       "              346,\n",
       "              347,\n",
       "              348,\n",
       "              349,\n",
       "              350,\n",
       "              351,\n",
       "              353,\n",
       "              356,\n",
       "              358,\n",
       "              359,\n",
       "              361,\n",
       "              362,\n",
       "              363,\n",
       "              364,\n",
       "              365,\n",
       "              366,\n",
       "              368,\n",
       "              370,\n",
       "              372,\n",
       "              374,\n",
       "              375,\n",
       "              376,\n",
       "              377,\n",
       "              378,\n",
       "              379,\n",
       "              380,\n",
       "              381,\n",
       "              382,\n",
       "              383,\n",
       "              386,\n",
       "              387,\n",
       "              388,\n",
       "              389,\n",
       "              393,\n",
       "              395,\n",
       "              399,\n",
       "              401,\n",
       "              402,\n",
       "              404,\n",
       "              405,\n",
       "              406,\n",
       "              408,\n",
       "              410,\n",
       "              412,\n",
       "              413,\n",
       "              414,\n",
       "              415,\n",
       "              416,\n",
       "              419,\n",
       "              420,\n",
       "              421,\n",
       "              423,\n",
       "              424,\n",
       "              425,\n",
       "              427,\n",
       "              429,\n",
       "              432,\n",
       "              433,\n",
       "              434,\n",
       "              435,\n",
       "              436,\n",
       "              438,\n",
       "              439,\n",
       "              443,\n",
       "              445,\n",
       "              446,\n",
       "              447,\n",
       "              449,\n",
       "              450,\n",
       "              452,\n",
       "              454,\n",
       "              455,\n",
       "              457,\n",
       "              458,\n",
       "              459,\n",
       "              463,\n",
       "              464,\n",
       "              465,\n",
       "              466,\n",
       "              468,\n",
       "              470,\n",
       "              472,\n",
       "              476,\n",
       "              477,\n",
       "              478,\n",
       "              479,\n",
       "              482,\n",
       "              483,\n",
       "              484,\n",
       "              485,\n",
       "              486,\n",
       "              487,\n",
       "              489,\n",
       "              491,\n",
       "              492,\n",
       "              493,\n",
       "              494,\n",
       "              499,\n",
       "              500,\n",
       "              502,\n",
       "              503,\n",
       "              505,\n",
       "              506,\n",
       "              508,\n",
       "              509,\n",
       "              510,\n",
       "              511,\n",
       "              514,\n",
       "              515,\n",
       "              516,\n",
       "              518,\n",
       "              519,\n",
       "              520,\n",
       "              521,\n",
       "              523,\n",
       "              527,\n",
       "              529,\n",
       "              530,\n",
       "              531,\n",
       "              532,\n",
       "              536,\n",
       "              540,\n",
       "              541,\n",
       "              543,\n",
       "              545,\n",
       "              546,\n",
       "              548,\n",
       "              549,\n",
       "              550,\n",
       "              551,\n",
       "              552,\n",
       "              553,\n",
       "              556,\n",
       "              557,\n",
       "              558,\n",
       "              559,\n",
       "              560,\n",
       "              561,\n",
       "              563,\n",
       "              565,\n",
       "              566,\n",
       "              567,\n",
       "              570,\n",
       "              571,\n",
       "              573,\n",
       "              576,\n",
       "              577,\n",
       "              581,\n",
       "              584,\n",
       "              585,\n",
       "              590,\n",
       "              592,\n",
       "              594,\n",
       "              595,\n",
       "              598,\n",
       "              601,\n",
       "              603,\n",
       "              604,\n",
       "              605,\n",
       "              606,\n",
       "              607,\n",
       "              608,\n",
       "              609,\n",
       "              611,\n",
       "              612,\n",
       "              613,\n",
       "              614,\n",
       "              615,\n",
       "              617,\n",
       "              618,\n",
       "              619,\n",
       "              621,\n",
       "              622,\n",
       "              623,\n",
       "              625,\n",
       "              626,\n",
       "              627,\n",
       "              628,\n",
       "              629,\n",
       "              630,\n",
       "              632,\n",
       "              638,\n",
       "              639,\n",
       "              640,\n",
       "              644,\n",
       "              645,\n",
       "              648,\n",
       "              649,\n",
       "              650,\n",
       "              653,\n",
       "              654,\n",
       "              656,\n",
       "              657,\n",
       "              658,\n",
       "              659,\n",
       "              660,\n",
       "              662,\n",
       "              666,\n",
       "              667,\n",
       "              668,\n",
       "              669,\n",
       "              670,\n",
       "              671,\n",
       "              673,\n",
       "              674,\n",
       "              676,\n",
       "              677,\n",
       "              678,\n",
       "              680,\n",
       "              681,\n",
       "              684,\n",
       "              685,\n",
       "              686,\n",
       "              687,\n",
       "              688,\n",
       "              689,\n",
       "              690,\n",
       "              691,\n",
       "              693,\n",
       "              694,\n",
       "              695,\n",
       "              696,\n",
       "              697,\n",
       "              698,\n",
       "              702,\n",
       "              703,\n",
       "              704,\n",
       "              705,\n",
       "              706,\n",
       "              708,\n",
       "              709,\n",
       "              710,\n",
       "              711,\n",
       "              712,\n",
       "              713,\n",
       "              714,\n",
       "              715,\n",
       "              716,\n",
       "              717,\n",
       "              718,\n",
       "              719,\n",
       "              720,\n",
       "              721,\n",
       "              722,\n",
       "              723,\n",
       "              725,\n",
       "              726,\n",
       "              727,\n",
       "              730,\n",
       "              731,\n",
       "              732,\n",
       "              733,\n",
       "              734,\n",
       "              735,\n",
       "              739,\n",
       "              742,\n",
       "              743,\n",
       "              745,\n",
       "              746,\n",
       "              748,\n",
       "              749,\n",
       "              752,\n",
       "              755,\n",
       "              756,\n",
       "              757,\n",
       "              758,\n",
       "              761,\n",
       "              763,\n",
       "              764,\n",
       "              766,\n",
       "              767,\n",
       "              768,\n",
       "              769,\n",
       "              770,\n",
       "              771,\n",
       "              772,\n",
       "              774,\n",
       "              777,\n",
       "              778,\n",
       "              782,\n",
       "              784,\n",
       "              785,\n",
       "              786,\n",
       "              787,\n",
       "              789,\n",
       "              791,\n",
       "              792,\n",
       "              794,\n",
       "              795,\n",
       "              798,\n",
       "              799,\n",
       "              800,\n",
       "              801,\n",
       "              803,\n",
       "              804,\n",
       "              805,\n",
       "              807,\n",
       "              809,\n",
       "              811,\n",
       "              813,\n",
       "              815,\n",
       "              816,\n",
       "              817,\n",
       "              818,\n",
       "              820,\n",
       "              823,\n",
       "              824,\n",
       "              828,\n",
       "              830,\n",
       "              831,\n",
       "              832,\n",
       "              835,\n",
       "              838,\n",
       "              839,\n",
       "              843,\n",
       "              844,\n",
       "              846,\n",
       "              847,\n",
       "              848,\n",
       "              849,\n",
       "              851,\n",
       "              853,\n",
       "              854,\n",
       "              855,\n",
       "              857,\n",
       "              858,\n",
       "              859,\n",
       "              860,\n",
       "              862,\n",
       "              864,\n",
       "              865,\n",
       "              866,\n",
       "              867,\n",
       "              868,\n",
       "              872,\n",
       "              874,\n",
       "              875,\n",
       "              878,\n",
       "              884,\n",
       "              886,\n",
       "              887,\n",
       "              888,\n",
       "              892,\n",
       "              893,\n",
       "              895,\n",
       "              896,\n",
       "              897,\n",
       "              898,\n",
       "              899,\n",
       "              901,\n",
       "              902,\n",
       "              903,\n",
       "              904,\n",
       "              905,\n",
       "              906,\n",
       "              908,\n",
       "              909,\n",
       "              911,\n",
       "              913,\n",
       "              914,\n",
       "              915,\n",
       "              917,\n",
       "              919,\n",
       "              920,\n",
       "              921,\n",
       "              927,\n",
       "              928,\n",
       "              931,\n",
       "              934,\n",
       "              935,\n",
       "              936,\n",
       "              939,\n",
       "              940,\n",
       "              941,\n",
       "              942,\n",
       "              943,\n",
       "              944,\n",
       "              945,\n",
       "              946,\n",
       "              948,\n",
       "              949,\n",
       "              950,\n",
       "              951,\n",
       "              952,\n",
       "              953,\n",
       "              954,\n",
       "              955,\n",
       "              956,\n",
       "              957,\n",
       "              959,\n",
       "              960,\n",
       "              961,\n",
       "              963,\n",
       "              969,\n",
       "              970,\n",
       "              972,\n",
       "              973,\n",
       "              974,\n",
       "              976,\n",
       "              977,\n",
       "              978,\n",
       "              979,\n",
       "              981,\n",
       "              982,\n",
       "              985,\n",
       "              988,\n",
       "              992,\n",
       "              993,\n",
       "              995,\n",
       "              996,\n",
       "              997,\n",
       "              1000,\n",
       "              1001,\n",
       "              1002,\n",
       "              1004,\n",
       "              1006,\n",
       "              1008,\n",
       "              1009,\n",
       "              1010,\n",
       "              1011,\n",
       "              1013,\n",
       "              1014,\n",
       "              1015,\n",
       "              1017,\n",
       "              1021,\n",
       "              1023,\n",
       "              1024,\n",
       "              1025,\n",
       "              1026,\n",
       "              1027,\n",
       "              1028,\n",
       "              1029,\n",
       "              1030,\n",
       "              1031,\n",
       "              1035,\n",
       "              1036,\n",
       "              1037,\n",
       "              1038,\n",
       "              1039,\n",
       "              1040,\n",
       "              1042,\n",
       "              1044,\n",
       "              1046,\n",
       "              1047,\n",
       "              1048,\n",
       "              1049,\n",
       "              1051,\n",
       "              1052,\n",
       "              1053,\n",
       "              1054,\n",
       "              1055,\n",
       "              1059,\n",
       "              1060,\n",
       "              1061,\n",
       "              1065,\n",
       "              1066,\n",
       "              1068,\n",
       "              1070,\n",
       "              1071,\n",
       "              1072,\n",
       "              1075,\n",
       "              1076,\n",
       "              1078,\n",
       "              1079,\n",
       "              1081,\n",
       "              1082,\n",
       "              1084,\n",
       "              1086,\n",
       "              1089,\n",
       "              1090,\n",
       "              1091,\n",
       "              1093,\n",
       "              1094,\n",
       "              1095,\n",
       "              1096,\n",
       "              1097,\n",
       "              1098,\n",
       "              1101,\n",
       "              1102,\n",
       "              1105,\n",
       "              1106,\n",
       "              1107,\n",
       "              1108,\n",
       "              1110,\n",
       "              1111,\n",
       "              1113,\n",
       "              1115,\n",
       "              1116,\n",
       "              1117,\n",
       "              1118,\n",
       "              1121,\n",
       "              1122,\n",
       "              1123,\n",
       "              1125,\n",
       "              1126,\n",
       "              1127,\n",
       "              1130,\n",
       "              1131,\n",
       "              1133,\n",
       "              1136,\n",
       "              1137,\n",
       "              1138,\n",
       "              1140,\n",
       "              1142,\n",
       "              1147,\n",
       "              1151,\n",
       "              1152,\n",
       "              1154,\n",
       "              1157,\n",
       "              1158,\n",
       "              1160,\n",
       "              1161,\n",
       "              1162,\n",
       "              1164,\n",
       "              1165,\n",
       "              1166,\n",
       "              1167,\n",
       "              1170,\n",
       "              1171,\n",
       "              1175,\n",
       "              1178,\n",
       "              1180,\n",
       "              1184,\n",
       "              1185,\n",
       "              1186,\n",
       "              1187,\n",
       "              1188,\n",
       "              1189,\n",
       "              1191,\n",
       "              1193,\n",
       "              1195,\n",
       "              1196,\n",
       "              1197,\n",
       "              1201,\n",
       "              1202,\n",
       "              1204,\n",
       "              1205,\n",
       "              1208,\n",
       "              1212,\n",
       "              1214,\n",
       "              1216,\n",
       "              1219,\n",
       "              1221,\n",
       "              1222,\n",
       "              1226,\n",
       "              1227,\n",
       "              1229,\n",
       "              1230,\n",
       "              1232,\n",
       "              1233,\n",
       "              1235,\n",
       "              1236,\n",
       "              1238,\n",
       "              1239,\n",
       "              1240,\n",
       "              1241,\n",
       "              1244,\n",
       "              1245,\n",
       "              1247,\n",
       "              1248,\n",
       "              1251,\n",
       "              1254,\n",
       "              1255,\n",
       "              1256,\n",
       "              1257,\n",
       "              1258,\n",
       "              1259,\n",
       "              1261,\n",
       "              1262,\n",
       "              1263,\n",
       "              1264,\n",
       "              1266,\n",
       "              1268,\n",
       "              1270,\n",
       "              1271,\n",
       "              1272,\n",
       "              1273,\n",
       "              1274,\n",
       "              1275,\n",
       "              1276,\n",
       "              1279,\n",
       "              1281,\n",
       "              1282,\n",
       "              1283,\n",
       "              1284,\n",
       "              1285,\n",
       "              1286,\n",
       "              1288,\n",
       "              1289,\n",
       "              1291,\n",
       "              1294,\n",
       "              1296,\n",
       "              1297,\n",
       "              1302,\n",
       "              1303,\n",
       "              1304,\n",
       "              1305,\n",
       "              1306,\n",
       "              1307,\n",
       "              1308,\n",
       "              1311,\n",
       "              1312,\n",
       "              1315,\n",
       "              1317,\n",
       "              1318,\n",
       "              1320,\n",
       "              1321,\n",
       "              1324,\n",
       "              1325,\n",
       "              1326,\n",
       "              1327,\n",
       "              1330,\n",
       "              1332,\n",
       "              1335,\n",
       "              1337,\n",
       "              1339,\n",
       "              1340,\n",
       "              1341,\n",
       "              1344,\n",
       "              1346,\n",
       "              1347,\n",
       "              1348,\n",
       "              1349,\n",
       "              1351,\n",
       "              1352,\n",
       "              1353,\n",
       "              1354,\n",
       "              1355,\n",
       "              1356,\n",
       "              1357,\n",
       "              1358,\n",
       "              1360,\n",
       "              1361,\n",
       "              1362,\n",
       "              1364,\n",
       "              1365,\n",
       "              1366,\n",
       "              1368,\n",
       "              1369,\n",
       "              1371,\n",
       "              1372,\n",
       "              1373,\n",
       "              1374,\n",
       "              1376,\n",
       "              1377,\n",
       "              1378,\n",
       "              1379,\n",
       "              1380,\n",
       "              1382,\n",
       "              1384,\n",
       "              1386,\n",
       "              1388,\n",
       "              1390,\n",
       "              1391,\n",
       "              1392,\n",
       "              1393,\n",
       "              1394,\n",
       "              1395,\n",
       "              1396,\n",
       "              1398,\n",
       "              1400,\n",
       "              1401,\n",
       "              1402,\n",
       "              1403,\n",
       "              1405,\n",
       "              1406,\n",
       "              1408,\n",
       "              1409,\n",
       "              1411,\n",
       "              1413,\n",
       "              1418,\n",
       "              1421,\n",
       "              1422,\n",
       "              1424,\n",
       "              1425,\n",
       "              1428,\n",
       "              1431,\n",
       "              1432,\n",
       "              1434,\n",
       "              1435,\n",
       "              1436,\n",
       "              1438,\n",
       "              1439,\n",
       "              1440,\n",
       "              1442,\n",
       "              1443,\n",
       "              1445,\n",
       "              1446,\n",
       "              1449,\n",
       "              1450,\n",
       "              1453,\n",
       "              1454,\n",
       "              1455,\n",
       "              1456,\n",
       "              1457,\n",
       "              1458,\n",
       "              1460,\n",
       "              1461,\n",
       "              1463,\n",
       "              1464,\n",
       "              1466,\n",
       "              1469,\n",
       "              1470,\n",
       "              1471,\n",
       "              1472,\n",
       "              1474,\n",
       "              1475,\n",
       "              1478,\n",
       "              1479,\n",
       "              1481,\n",
       "              1482,\n",
       "              1483,\n",
       "              1484,\n",
       "              1485,\n",
       "              1486,\n",
       "              1488,\n",
       "              1489,\n",
       "              1490,\n",
       "              1492,\n",
       "              1493,\n",
       "              1494,\n",
       "              1495,\n",
       "              1496,\n",
       "              1498,\n",
       "              1499,\n",
       "              1500,\n",
       "              1502,\n",
       "              1507,\n",
       "              1509,\n",
       "              1510,\n",
       "              1511,\n",
       "              1513,\n",
       "              1515,\n",
       "              1516,\n",
       "              1521,\n",
       "              1522,\n",
       "              1523,\n",
       "              1525,\n",
       "              1526,\n",
       "              1528,\n",
       "              1529,\n",
       "              1531,\n",
       "              1532,\n",
       "              1533,\n",
       "              1534,\n",
       "              1537,\n",
       "              1539,\n",
       "              1540,\n",
       "              1543,\n",
       "              1544,\n",
       "              1548,\n",
       "              1549,\n",
       "              1551,\n",
       "              1552,\n",
       "              1553,\n",
       "              1555,\n",
       "              1556,\n",
       "              1557,\n",
       "              1558,\n",
       "              1559,\n",
       "              1563,\n",
       "              1566,\n",
       "              1567,\n",
       "              1569,\n",
       "              1570,\n",
       "              1571,\n",
       "              1572,\n",
       "              1573,\n",
       "              1574,\n",
       "              1576,\n",
       "              1577,\n",
       "              1579,\n",
       "              1582,\n",
       "              1583,\n",
       "              1585,\n",
       "              1586,\n",
       "              1587,\n",
       "              1588},\n",
       "             'male': {0,\n",
       "              1,\n",
       "              5,\n",
       "              13,\n",
       "              14,\n",
       "              15,\n",
       "              16,\n",
       "              18,\n",
       "              20,\n",
       "              26,\n",
       "              27,\n",
       "              30,\n",
       "              31,\n",
       "              35,\n",
       "              36,\n",
       "              37,\n",
       "              39,\n",
       "              42,\n",
       "              43,\n",
       "              45,\n",
       "              46,\n",
       "              49,\n",
       "              52,\n",
       "              54,\n",
       "              55,\n",
       "              58,\n",
       "              59,\n",
       "              63,\n",
       "              64,\n",
       "              65,\n",
       "              71,\n",
       "              73,\n",
       "              76,\n",
       "              77,\n",
       "              81,\n",
       "              84,\n",
       "              86,\n",
       "              94,\n",
       "              97,\n",
       "              99,\n",
       "              100,\n",
       "              101,\n",
       "              102,\n",
       "              103,\n",
       "              105,\n",
       "              106,\n",
       "              110,\n",
       "              112,\n",
       "              118,\n",
       "              133,\n",
       "              142,\n",
       "              144,\n",
       "              145,\n",
       "              146,\n",
       "              147,\n",
       "              148,\n",
       "              149,\n",
       "              151,\n",
       "              152,\n",
       "              155,\n",
       "              156,\n",
       "              158,\n",
       "              161,\n",
       "              162,\n",
       "              165,\n",
       "              166,\n",
       "              168,\n",
       "              170,\n",
       "              174,\n",
       "              175,\n",
       "              180,\n",
       "              184,\n",
       "              188,\n",
       "              189,\n",
       "              191,\n",
       "              195,\n",
       "              196,\n",
       "              203,\n",
       "              206,\n",
       "              207,\n",
       "              209,\n",
       "              220,\n",
       "              223,\n",
       "              226,\n",
       "              227,\n",
       "              229,\n",
       "              233,\n",
       "              238,\n",
       "              239,\n",
       "              251,\n",
       "              252,\n",
       "              253,\n",
       "              254,\n",
       "              256,\n",
       "              259,\n",
       "              261,\n",
       "              263,\n",
       "              265,\n",
       "              266,\n",
       "              267,\n",
       "              268,\n",
       "              269,\n",
       "              271,\n",
       "              272,\n",
       "              273,\n",
       "              275,\n",
       "              276,\n",
       "              284,\n",
       "              286,\n",
       "              294,\n",
       "              296,\n",
       "              297,\n",
       "              300,\n",
       "              301,\n",
       "              305,\n",
       "              307,\n",
       "              308,\n",
       "              311,\n",
       "              315,\n",
       "              316,\n",
       "              318,\n",
       "              323,\n",
       "              327,\n",
       "              328,\n",
       "              330,\n",
       "              331,\n",
       "              332,\n",
       "              335,\n",
       "              336,\n",
       "              338,\n",
       "              344,\n",
       "              352,\n",
       "              354,\n",
       "              355,\n",
       "              357,\n",
       "              360,\n",
       "              367,\n",
       "              369,\n",
       "              371,\n",
       "              373,\n",
       "              384,\n",
       "              385,\n",
       "              390,\n",
       "              391,\n",
       "              392,\n",
       "              394,\n",
       "              396,\n",
       "              397,\n",
       "              398,\n",
       "              400,\n",
       "              403,\n",
       "              407,\n",
       "              409,\n",
       "              411,\n",
       "              417,\n",
       "              418,\n",
       "              422,\n",
       "              426,\n",
       "              428,\n",
       "              430,\n",
       "              431,\n",
       "              437,\n",
       "              440,\n",
       "              441,\n",
       "              442,\n",
       "              444,\n",
       "              448,\n",
       "              451,\n",
       "              453,\n",
       "              456,\n",
       "              460,\n",
       "              461,\n",
       "              462,\n",
       "              467,\n",
       "              469,\n",
       "              471,\n",
       "              473,\n",
       "              474,\n",
       "              475,\n",
       "              480,\n",
       "              481,\n",
       "              488,\n",
       "              490,\n",
       "              495,\n",
       "              496,\n",
       "              497,\n",
       "              498,\n",
       "              501,\n",
       "              504,\n",
       "              507,\n",
       "              512,\n",
       "              513,\n",
       "              517,\n",
       "              522,\n",
       "              524,\n",
       "              525,\n",
       "              526,\n",
       "              528,\n",
       "              533,\n",
       "              534,\n",
       "              535,\n",
       "              537,\n",
       "              538,\n",
       "              539,\n",
       "              542,\n",
       "              544,\n",
       "              547,\n",
       "              554,\n",
       "              555,\n",
       "              562,\n",
       "              564,\n",
       "              568,\n",
       "              569,\n",
       "              572,\n",
       "              574,\n",
       "              575,\n",
       "              578,\n",
       "              579,\n",
       "              580,\n",
       "              582,\n",
       "              583,\n",
       "              586,\n",
       "              587,\n",
       "              588,\n",
       "              589,\n",
       "              591,\n",
       "              593,\n",
       "              596,\n",
       "              597,\n",
       "              599,\n",
       "              600,\n",
       "              602,\n",
       "              610,\n",
       "              616,\n",
       "              620,\n",
       "              624,\n",
       "              631,\n",
       "              633,\n",
       "              634,\n",
       "              635,\n",
       "              636,\n",
       "              637,\n",
       "              641,\n",
       "              642,\n",
       "              643,\n",
       "              646,\n",
       "              647,\n",
       "              651,\n",
       "              652,\n",
       "              655,\n",
       "              661,\n",
       "              663,\n",
       "              664,\n",
       "              665,\n",
       "              672,\n",
       "              675,\n",
       "              679,\n",
       "              682,\n",
       "              683,\n",
       "              692,\n",
       "              699,\n",
       "              700,\n",
       "              701,\n",
       "              707,\n",
       "              724,\n",
       "              728,\n",
       "              729,\n",
       "              736,\n",
       "              737,\n",
       "              738,\n",
       "              740,\n",
       "              741,\n",
       "              744,\n",
       "              747,\n",
       "              750,\n",
       "              751,\n",
       "              753,\n",
       "              754,\n",
       "              759,\n",
       "              760,\n",
       "              762,\n",
       "              765,\n",
       "              773,\n",
       "              775,\n",
       "              776,\n",
       "              779,\n",
       "              780,\n",
       "              781,\n",
       "              783,\n",
       "              788,\n",
       "              790,\n",
       "              793,\n",
       "              796,\n",
       "              797,\n",
       "              802,\n",
       "              806,\n",
       "              808,\n",
       "              810,\n",
       "              812,\n",
       "              814,\n",
       "              819,\n",
       "              821,\n",
       "              822,\n",
       "              825,\n",
       "              826,\n",
       "              827,\n",
       "              829,\n",
       "              833,\n",
       "              834,\n",
       "              836,\n",
       "              837,\n",
       "              840,\n",
       "              841,\n",
       "              842,\n",
       "              845,\n",
       "              850,\n",
       "              852,\n",
       "              856,\n",
       "              861,\n",
       "              863,\n",
       "              869,\n",
       "              870,\n",
       "              871,\n",
       "              873,\n",
       "              876,\n",
       "              877,\n",
       "              879,\n",
       "              880,\n",
       "              881,\n",
       "              882,\n",
       "              883,\n",
       "              885,\n",
       "              889,\n",
       "              890,\n",
       "              891,\n",
       "              894,\n",
       "              900,\n",
       "              907,\n",
       "              910,\n",
       "              912,\n",
       "              916,\n",
       "              918,\n",
       "              922,\n",
       "              923,\n",
       "              924,\n",
       "              925,\n",
       "              926,\n",
       "              929,\n",
       "              930,\n",
       "              932,\n",
       "              933,\n",
       "              937,\n",
       "              938,\n",
       "              947,\n",
       "              958,\n",
       "              962,\n",
       "              964,\n",
       "              965,\n",
       "              966,\n",
       "              967,\n",
       "              968,\n",
       "              971,\n",
       "              975,\n",
       "              980,\n",
       "              983,\n",
       "              984,\n",
       "              986,\n",
       "              987,\n",
       "              989,\n",
       "              990,\n",
       "              991,\n",
       "              994,\n",
       "              998,\n",
       "              999,\n",
       "              1003,\n",
       "              1005,\n",
       "              1007,\n",
       "              1012,\n",
       "              1016,\n",
       "              1018,\n",
       "              1019,\n",
       "              1020,\n",
       "              1022,\n",
       "              1032,\n",
       "              1033,\n",
       "              1034,\n",
       "              1041,\n",
       "              1043,\n",
       "              1045,\n",
       "              1050,\n",
       "              1056,\n",
       "              1057,\n",
       "              1058,\n",
       "              1062,\n",
       "              1063,\n",
       "              1064,\n",
       "              1067,\n",
       "              1069,\n",
       "              1073,\n",
       "              1074,\n",
       "              1077,\n",
       "              1080,\n",
       "              1083,\n",
       "              1085,\n",
       "              1087,\n",
       "              1088,\n",
       "              1092,\n",
       "              1099,\n",
       "              1100,\n",
       "              1103,\n",
       "              1104,\n",
       "              1109,\n",
       "              1112,\n",
       "              1114,\n",
       "              1119,\n",
       "              1120,\n",
       "              1124,\n",
       "              1128,\n",
       "              1129,\n",
       "              1132,\n",
       "              1134,\n",
       "              1135,\n",
       "              1139,\n",
       "              1141,\n",
       "              1143,\n",
       "              1144,\n",
       "              1145,\n",
       "              1146,\n",
       "              1148,\n",
       "              1149,\n",
       "              1150,\n",
       "              1153,\n",
       "              1155,\n",
       "              1156,\n",
       "              1159,\n",
       "              1163,\n",
       "              1168,\n",
       "              1169,\n",
       "              1172,\n",
       "              1173,\n",
       "              1174,\n",
       "              1176,\n",
       "              1177,\n",
       "              1179,\n",
       "              1181,\n",
       "              1182,\n",
       "              1183,\n",
       "              1190,\n",
       "              1192,\n",
       "              1194,\n",
       "              1198,\n",
       "              1199,\n",
       "              1200,\n",
       "              1203,\n",
       "              1206,\n",
       "              1207,\n",
       "              1209,\n",
       "              1210,\n",
       "              1211,\n",
       "              1213,\n",
       "              1215,\n",
       "              1217,\n",
       "              1218,\n",
       "              1220,\n",
       "              1223,\n",
       "              1224,\n",
       "              1225,\n",
       "              1228,\n",
       "              1231,\n",
       "              1234,\n",
       "              1237,\n",
       "              1242,\n",
       "              1243,\n",
       "              1246,\n",
       "              1249,\n",
       "              1250,\n",
       "              1252,\n",
       "              1253,\n",
       "              1260,\n",
       "              1265,\n",
       "              1267,\n",
       "              1269,\n",
       "              1277,\n",
       "              1278,\n",
       "              1280,\n",
       "              1287,\n",
       "              1290,\n",
       "              1292,\n",
       "              1293,\n",
       "              1295,\n",
       "              1298,\n",
       "              1299,\n",
       "              1300,\n",
       "              1301,\n",
       "              1309,\n",
       "              1310,\n",
       "              1313,\n",
       "              1314,\n",
       "              1316,\n",
       "              1319,\n",
       "              1322,\n",
       "              1323,\n",
       "              1328,\n",
       "              1329,\n",
       "              1331,\n",
       "              1333,\n",
       "              1334,\n",
       "              1336,\n",
       "              1338,\n",
       "              1342,\n",
       "              1343,\n",
       "              1345,\n",
       "              1350,\n",
       "              1359,\n",
       "              1363,\n",
       "              1367,\n",
       "              1370,\n",
       "              1375,\n",
       "              1381,\n",
       "              1383,\n",
       "              1385,\n",
       "              1387,\n",
       "              1389,\n",
       "              1397,\n",
       "              1399,\n",
       "              1404,\n",
       "              1407,\n",
       "              1410,\n",
       "              1412,\n",
       "              1414,\n",
       "              1415,\n",
       "              1416,\n",
       "              1417,\n",
       "              1419,\n",
       "              1420,\n",
       "              1423,\n",
       "              1426,\n",
       "              1427,\n",
       "              1429,\n",
       "              1430,\n",
       "              1433,\n",
       "              1437,\n",
       "              1441,\n",
       "              1444,\n",
       "              1447,\n",
       "              1448,\n",
       "              1451,\n",
       "              1452,\n",
       "              1459,\n",
       "              1462,\n",
       "              1465,\n",
       "              1467,\n",
       "              1468,\n",
       "              1473,\n",
       "              1476,\n",
       "              1477,\n",
       "              1480,\n",
       "              1487,\n",
       "              1491,\n",
       "              1497,\n",
       "              1501,\n",
       "              1503,\n",
       "              1504,\n",
       "              1505,\n",
       "              1506,\n",
       "              1508,\n",
       "              1512,\n",
       "              1514,\n",
       "              1517,\n",
       "              1518,\n",
       "              1519,\n",
       "              1520,\n",
       "              1524,\n",
       "              1527,\n",
       "              1530,\n",
       "              1535,\n",
       "              1536,\n",
       "              1538,\n",
       "              1541,\n",
       "              1542,\n",
       "              1545,\n",
       "              1546,\n",
       "              1547,\n",
       "              1550,\n",
       "              1554,\n",
       "              1560,\n",
       "              1561,\n",
       "              1562,\n",
       "              1564,\n",
       "              1565,\n",
       "              1568,\n",
       "              1575,\n",
       "              1578,\n",
       "              1580,\n",
       "              1581,\n",
       "              1584}})"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "refsets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defaultdict(set,\n",
       "            {'female': {1,\n",
       "              2,\n",
       "              3,\n",
       "              4,\n",
       "              6,\n",
       "              8,\n",
       "              10,\n",
       "              12,\n",
       "              13,\n",
       "              17,\n",
       "              20,\n",
       "              21,\n",
       "              22,\n",
       "              24,\n",
       "              25,\n",
       "              26,\n",
       "              27,\n",
       "              28,\n",
       "              29,\n",
       "              31,\n",
       "              32,\n",
       "              33,\n",
       "              34,\n",
       "              36,\n",
       "              38,\n",
       "              40,\n",
       "              41,\n",
       "              43,\n",
       "              46,\n",
       "              47,\n",
       "              50,\n",
       "              52,\n",
       "              53,\n",
       "              54,\n",
       "              57,\n",
       "              60,\n",
       "              61,\n",
       "              63,\n",
       "              64,\n",
       "              65,\n",
       "              66,\n",
       "              67,\n",
       "              68,\n",
       "              69,\n",
       "              70,\n",
       "              71,\n",
       "              72,\n",
       "              74,\n",
       "              77,\n",
       "              79,\n",
       "              80,\n",
       "              82,\n",
       "              85,\n",
       "              87,\n",
       "              88,\n",
       "              89,\n",
       "              90,\n",
       "              91,\n",
       "              92,\n",
       "              93,\n",
       "              95,\n",
       "              96,\n",
       "              97,\n",
       "              99,\n",
       "              101,\n",
       "              102,\n",
       "              104,\n",
       "              105,\n",
       "              107,\n",
       "              108,\n",
       "              109,\n",
       "              111,\n",
       "              112,\n",
       "              113,\n",
       "              114,\n",
       "              115,\n",
       "              117,\n",
       "              118,\n",
       "              119,\n",
       "              120,\n",
       "              121,\n",
       "              122,\n",
       "              123,\n",
       "              124,\n",
       "              125,\n",
       "              126,\n",
       "              127,\n",
       "              129,\n",
       "              130,\n",
       "              131,\n",
       "              132,\n",
       "              134,\n",
       "              135,\n",
       "              136,\n",
       "              137,\n",
       "              138,\n",
       "              139,\n",
       "              140,\n",
       "              141,\n",
       "              143,\n",
       "              145,\n",
       "              147,\n",
       "              148,\n",
       "              149,\n",
       "              150,\n",
       "              151,\n",
       "              152,\n",
       "              154,\n",
       "              155,\n",
       "              157,\n",
       "              159,\n",
       "              160,\n",
       "              161,\n",
       "              162,\n",
       "              163,\n",
       "              164,\n",
       "              165,\n",
       "              166,\n",
       "              167,\n",
       "              168,\n",
       "              170,\n",
       "              171,\n",
       "              172,\n",
       "              173,\n",
       "              174,\n",
       "              176,\n",
       "              177,\n",
       "              178,\n",
       "              179,\n",
       "              180,\n",
       "              181,\n",
       "              182,\n",
       "              183,\n",
       "              184,\n",
       "              185,\n",
       "              186,\n",
       "              187,\n",
       "              190,\n",
       "              191,\n",
       "              192,\n",
       "              193,\n",
       "              194,\n",
       "              197,\n",
       "              198,\n",
       "              199,\n",
       "              200,\n",
       "              201,\n",
       "              202,\n",
       "              204,\n",
       "              208,\n",
       "              210,\n",
       "              211,\n",
       "              212,\n",
       "              213,\n",
       "              214,\n",
       "              215,\n",
       "              216,\n",
       "              217,\n",
       "              218,\n",
       "              219,\n",
       "              220,\n",
       "              221,\n",
       "              222,\n",
       "              223,\n",
       "              224,\n",
       "              226,\n",
       "              227,\n",
       "              228,\n",
       "              229,\n",
       "              230,\n",
       "              231,\n",
       "              232,\n",
       "              234,\n",
       "              235,\n",
       "              236,\n",
       "              237,\n",
       "              238,\n",
       "              240,\n",
       "              241,\n",
       "              243,\n",
       "              244,\n",
       "              245,\n",
       "              246,\n",
       "              247,\n",
       "              248,\n",
       "              250,\n",
       "              252,\n",
       "              254,\n",
       "              255,\n",
       "              256,\n",
       "              257,\n",
       "              258,\n",
       "              262,\n",
       "              264,\n",
       "              267,\n",
       "              270,\n",
       "              271,\n",
       "              273,\n",
       "              274,\n",
       "              277,\n",
       "              278,\n",
       "              279,\n",
       "              280,\n",
       "              281,\n",
       "              282,\n",
       "              283,\n",
       "              284,\n",
       "              285,\n",
       "              287,\n",
       "              288,\n",
       "              289,\n",
       "              290,\n",
       "              291,\n",
       "              293,\n",
       "              295,\n",
       "              297,\n",
       "              298,\n",
       "              299,\n",
       "              302,\n",
       "              303,\n",
       "              304,\n",
       "              305,\n",
       "              306,\n",
       "              309,\n",
       "              310,\n",
       "              312,\n",
       "              313,\n",
       "              314,\n",
       "              315,\n",
       "              317,\n",
       "              320,\n",
       "              321,\n",
       "              322,\n",
       "              324,\n",
       "              325,\n",
       "              326,\n",
       "              333,\n",
       "              334,\n",
       "              337,\n",
       "              338,\n",
       "              339,\n",
       "              340,\n",
       "              341,\n",
       "              342,\n",
       "              343,\n",
       "              346,\n",
       "              347,\n",
       "              348,\n",
       "              349,\n",
       "              350,\n",
       "              351,\n",
       "              353,\n",
       "              355,\n",
       "              356,\n",
       "              358,\n",
       "              359,\n",
       "              361,\n",
       "              362,\n",
       "              363,\n",
       "              364,\n",
       "              365,\n",
       "              366,\n",
       "              367,\n",
       "              368,\n",
       "              370,\n",
       "              372,\n",
       "              377,\n",
       "              378,\n",
       "              379,\n",
       "              380,\n",
       "              381,\n",
       "              383,\n",
       "              385,\n",
       "              388,\n",
       "              389,\n",
       "              392,\n",
       "              394,\n",
       "              395,\n",
       "              398,\n",
       "              399,\n",
       "              401,\n",
       "              402,\n",
       "              404,\n",
       "              405,\n",
       "              406,\n",
       "              407,\n",
       "              408,\n",
       "              409,\n",
       "              410,\n",
       "              412,\n",
       "              413,\n",
       "              414,\n",
       "              415,\n",
       "              416,\n",
       "              418,\n",
       "              419,\n",
       "              420,\n",
       "              421,\n",
       "              423,\n",
       "              424,\n",
       "              425,\n",
       "              427,\n",
       "              428,\n",
       "              429,\n",
       "              432,\n",
       "              433,\n",
       "              434,\n",
       "              435,\n",
       "              436,\n",
       "              438,\n",
       "              439,\n",
       "              443,\n",
       "              445,\n",
       "              446,\n",
       "              447,\n",
       "              449,\n",
       "              450,\n",
       "              453,\n",
       "              454,\n",
       "              455,\n",
       "              456,\n",
       "              457,\n",
       "              458,\n",
       "              459,\n",
       "              460,\n",
       "              463,\n",
       "              464,\n",
       "              465,\n",
       "              466,\n",
       "              468,\n",
       "              470,\n",
       "              472,\n",
       "              473,\n",
       "              474,\n",
       "              476,\n",
       "              478,\n",
       "              479,\n",
       "              480,\n",
       "              482,\n",
       "              483,\n",
       "              484,\n",
       "              485,\n",
       "              486,\n",
       "              487,\n",
       "              489,\n",
       "              491,\n",
       "              492,\n",
       "              493,\n",
       "              494,\n",
       "              495,\n",
       "              500,\n",
       "              501,\n",
       "              502,\n",
       "              503,\n",
       "              504,\n",
       "              505,\n",
       "              506,\n",
       "              507,\n",
       "              508,\n",
       "              509,\n",
       "              510,\n",
       "              511,\n",
       "              514,\n",
       "              515,\n",
       "              516,\n",
       "              518,\n",
       "              519,\n",
       "              520,\n",
       "              527,\n",
       "              529,\n",
       "              530,\n",
       "              531,\n",
       "              532,\n",
       "              535,\n",
       "              536,\n",
       "              537,\n",
       "              538,\n",
       "              539,\n",
       "              540,\n",
       "              541,\n",
       "              543,\n",
       "              545,\n",
       "              546,\n",
       "              547,\n",
       "              548,\n",
       "              549,\n",
       "              550,\n",
       "              551,\n",
       "              552,\n",
       "              553,\n",
       "              555,\n",
       "              556,\n",
       "              557,\n",
       "              558,\n",
       "              559,\n",
       "              560,\n",
       "              561,\n",
       "              566,\n",
       "              567,\n",
       "              570,\n",
       "              571,\n",
       "              573,\n",
       "              577,\n",
       "              578,\n",
       "              581,\n",
       "              584,\n",
       "              586,\n",
       "              589,\n",
       "              590,\n",
       "              591,\n",
       "              592,\n",
       "              593,\n",
       "              594,\n",
       "              595,\n",
       "              596,\n",
       "              598,\n",
       "              601,\n",
       "              603,\n",
       "              604,\n",
       "              605,\n",
       "              606,\n",
       "              607,\n",
       "              608,\n",
       "              612,\n",
       "              613,\n",
       "              614,\n",
       "              615,\n",
       "              616,\n",
       "              617,\n",
       "              618,\n",
       "              619,\n",
       "              620,\n",
       "              621,\n",
       "              622,\n",
       "              623,\n",
       "              625,\n",
       "              626,\n",
       "              627,\n",
       "              628,\n",
       "              629,\n",
       "              630,\n",
       "              631,\n",
       "              632,\n",
       "              634,\n",
       "              635,\n",
       "              636,\n",
       "              637,\n",
       "              638,\n",
       "              639,\n",
       "              640,\n",
       "              643,\n",
       "              644,\n",
       "              645,\n",
       "              647,\n",
       "              648,\n",
       "              649,\n",
       "              652,\n",
       "              653,\n",
       "              654,\n",
       "              655,\n",
       "              656,\n",
       "              658,\n",
       "              659,\n",
       "              660,\n",
       "              661,\n",
       "              662,\n",
       "              664,\n",
       "              666,\n",
       "              667,\n",
       "              668,\n",
       "              669,\n",
       "              670,\n",
       "              671,\n",
       "              672,\n",
       "              673,\n",
       "              674,\n",
       "              676,\n",
       "              677,\n",
       "              678,\n",
       "              680,\n",
       "              681,\n",
       "              684,\n",
       "              685,\n",
       "              686,\n",
       "              687,\n",
       "              688,\n",
       "              690,\n",
       "              691,\n",
       "              692,\n",
       "              693,\n",
       "              694,\n",
       "              695,\n",
       "              696,\n",
       "              697,\n",
       "              701,\n",
       "              702,\n",
       "              703,\n",
       "              704,\n",
       "              707,\n",
       "              708,\n",
       "              710,\n",
       "              711,\n",
       "              712,\n",
       "              714,\n",
       "              715,\n",
       "              717,\n",
       "              720,\n",
       "              721,\n",
       "              722,\n",
       "              723,\n",
       "              724,\n",
       "              725,\n",
       "              726,\n",
       "              727,\n",
       "              728,\n",
       "              730,\n",
       "              731,\n",
       "              733,\n",
       "              734,\n",
       "              736,\n",
       "              739,\n",
       "              741,\n",
       "              742,\n",
       "              743,\n",
       "              745,\n",
       "              746,\n",
       "              747,\n",
       "              748,\n",
       "              749,\n",
       "              752,\n",
       "              753,\n",
       "              754,\n",
       "              755,\n",
       "              756,\n",
       "              757,\n",
       "              758,\n",
       "              759,\n",
       "              760,\n",
       "              761,\n",
       "              763,\n",
       "              764,\n",
       "              766,\n",
       "              768,\n",
       "              769,\n",
       "              770,\n",
       "              771,\n",
       "              772,\n",
       "              773,\n",
       "              775,\n",
       "              776,\n",
       "              777,\n",
       "              778,\n",
       "              779,\n",
       "              780,\n",
       "              782,\n",
       "              784,\n",
       "              785,\n",
       "              786,\n",
       "              787,\n",
       "              792,\n",
       "              794,\n",
       "              795,\n",
       "              799,\n",
       "              800,\n",
       "              801,\n",
       "              805,\n",
       "              807,\n",
       "              808,\n",
       "              809,\n",
       "              810,\n",
       "              811,\n",
       "              813,\n",
       "              815,\n",
       "              816,\n",
       "              817,\n",
       "              818,\n",
       "              820,\n",
       "              822,\n",
       "              823,\n",
       "              824,\n",
       "              827,\n",
       "              828,\n",
       "              829,\n",
       "              831,\n",
       "              832,\n",
       "              838,\n",
       "              839,\n",
       "              842,\n",
       "              844,\n",
       "              845,\n",
       "              846,\n",
       "              847,\n",
       "              848,\n",
       "              849,\n",
       "              851,\n",
       "              853,\n",
       "              855,\n",
       "              856,\n",
       "              857,\n",
       "              858,\n",
       "              859,\n",
       "              860,\n",
       "              861,\n",
       "              862,\n",
       "              864,\n",
       "              865,\n",
       "              866,\n",
       "              867,\n",
       "              869,\n",
       "              871,\n",
       "              872,\n",
       "              873,\n",
       "              874,\n",
       "              878,\n",
       "              883,\n",
       "              884,\n",
       "              886,\n",
       "              887,\n",
       "              888,\n",
       "              890,\n",
       "              891,\n",
       "              892,\n",
       "              893,\n",
       "              896,\n",
       "              897,\n",
       "              898,\n",
       "              899,\n",
       "              901,\n",
       "              902,\n",
       "              904,\n",
       "              905,\n",
       "              906,\n",
       "              908,\n",
       "              909,\n",
       "              910,\n",
       "              911,\n",
       "              912,\n",
       "              914,\n",
       "              915,\n",
       "              916,\n",
       "              917,\n",
       "              918,\n",
       "              919,\n",
       "              920,\n",
       "              921,\n",
       "              922,\n",
       "              923,\n",
       "              925,\n",
       "              926,\n",
       "              927,\n",
       "              928,\n",
       "              931,\n",
       "              932,\n",
       "              934,\n",
       "              935,\n",
       "              936,\n",
       "              939,\n",
       "              940,\n",
       "              941,\n",
       "              942,\n",
       "              943,\n",
       "              945,\n",
       "              946,\n",
       "              947,\n",
       "              948,\n",
       "              949,\n",
       "              950,\n",
       "              951,\n",
       "              952,\n",
       "              953,\n",
       "              955,\n",
       "              956,\n",
       "              957,\n",
       "              959,\n",
       "              960,\n",
       "              961,\n",
       "              963,\n",
       "              966,\n",
       "              967,\n",
       "              969,\n",
       "              972,\n",
       "              973,\n",
       "              974,\n",
       "              976,\n",
       "              977,\n",
       "              978,\n",
       "              979,\n",
       "              983,\n",
       "              985,\n",
       "              987,\n",
       "              988,\n",
       "              990,\n",
       "              993,\n",
       "              995,\n",
       "              996,\n",
       "              997,\n",
       "              999,\n",
       "              1000,\n",
       "              1001,\n",
       "              1003,\n",
       "              1004,\n",
       "              1005,\n",
       "              1006,\n",
       "              1008,\n",
       "              1009,\n",
       "              1010,\n",
       "              1011,\n",
       "              1013,\n",
       "              1014,\n",
       "              1016,\n",
       "              1017,\n",
       "              1021,\n",
       "              1022,\n",
       "              1023,\n",
       "              1024,\n",
       "              1025,\n",
       "              1026,\n",
       "              1028,\n",
       "              1030,\n",
       "              1031,\n",
       "              1034,\n",
       "              1035,\n",
       "              1036,\n",
       "              1037,\n",
       "              1038,\n",
       "              1039,\n",
       "              1040,\n",
       "              1042,\n",
       "              1044,\n",
       "              1046,\n",
       "              1047,\n",
       "              1048,\n",
       "              1049,\n",
       "              1051,\n",
       "              1054,\n",
       "              1055,\n",
       "              1056,\n",
       "              1057,\n",
       "              1059,\n",
       "              1060,\n",
       "              1062,\n",
       "              1063,\n",
       "              1064,\n",
       "              1065,\n",
       "              1066,\n",
       "              1067,\n",
       "              1068,\n",
       "              1069,\n",
       "              1070,\n",
       "              1071,\n",
       "              1072,\n",
       "              1073,\n",
       "              1074,\n",
       "              1075,\n",
       "              1076,\n",
       "              1077,\n",
       "              1078,\n",
       "              1079,\n",
       "              1080,\n",
       "              1081,\n",
       "              1082,\n",
       "              1083,\n",
       "              1084,\n",
       "              1086,\n",
       "              1087,\n",
       "              1089,\n",
       "              1090,\n",
       "              1091,\n",
       "              1093,\n",
       "              1094,\n",
       "              1095,\n",
       "              1096,\n",
       "              1098,\n",
       "              1101,\n",
       "              1102,\n",
       "              1104,\n",
       "              1105,\n",
       "              1107,\n",
       "              1108,\n",
       "              1110,\n",
       "              1111,\n",
       "              1113,\n",
       "              1116,\n",
       "              1117,\n",
       "              1118,\n",
       "              1121,\n",
       "              1122,\n",
       "              1125,\n",
       "              1126,\n",
       "              1127,\n",
       "              1128,\n",
       "              1130,\n",
       "              1131,\n",
       "              1132,\n",
       "              1133,\n",
       "              1136,\n",
       "              1137,\n",
       "              1138,\n",
       "              1139,\n",
       "              1140,\n",
       "              1141,\n",
       "              1142,\n",
       "              1143,\n",
       "              1144,\n",
       "              1147,\n",
       "              1148,\n",
       "              1151,\n",
       "              1152,\n",
       "              1153,\n",
       "              1154,\n",
       "              1157,\n",
       "              1158,\n",
       "              1160,\n",
       "              1161,\n",
       "              1162,\n",
       "              1164,\n",
       "              1165,\n",
       "              1167,\n",
       "              1169,\n",
       "              1170,\n",
       "              1171,\n",
       "              1174,\n",
       "              1175,\n",
       "              1176,\n",
       "              1177,\n",
       "              1178,\n",
       "              1180,\n",
       "              1181,\n",
       "              1182,\n",
       "              1184,\n",
       "              1185,\n",
       "              1186,\n",
       "              1189,\n",
       "              1190,\n",
       "              1191,\n",
       "              1193,\n",
       "              1194,\n",
       "              1195,\n",
       "              1196,\n",
       "              1197,\n",
       "              1198,\n",
       "              1201,\n",
       "              1202,\n",
       "              1204,\n",
       "              1205,\n",
       "              1208,\n",
       "              1211,\n",
       "              1212,\n",
       "              1214,\n",
       "              1215,\n",
       "              1216,\n",
       "              1219,\n",
       "              1220,\n",
       "              1221,\n",
       "              1222,\n",
       "              1223,\n",
       "              1226,\n",
       "              1227,\n",
       "              1228,\n",
       "              1229,\n",
       "              1230,\n",
       "              1233,\n",
       "              1234,\n",
       "              1235,\n",
       "              1238,\n",
       "              1239,\n",
       "              1241,\n",
       "              1242,\n",
       "              1244,\n",
       "              1245,\n",
       "              1247,\n",
       "              1248,\n",
       "              1250,\n",
       "              1251,\n",
       "              1253,\n",
       "              1254,\n",
       "              1255,\n",
       "              1256,\n",
       "              1257,\n",
       "              1258,\n",
       "              1259,\n",
       "              1261,\n",
       "              1263,\n",
       "              1265,\n",
       "              1266,\n",
       "              1267,\n",
       "              1268,\n",
       "              1269,\n",
       "              1270,\n",
       "              1271,\n",
       "              1272,\n",
       "              1273,\n",
       "              1274,\n",
       "              1275,\n",
       "              1276,\n",
       "              1279,\n",
       "              1280,\n",
       "              1281,\n",
       "              1282,\n",
       "              1283,\n",
       "              1284,\n",
       "              1285,\n",
       "              1286,\n",
       "              1288,\n",
       "              1289,\n",
       "              1290,\n",
       "              1291,\n",
       "              1293,\n",
       "              1294,\n",
       "              1295,\n",
       "              1296,\n",
       "              1297,\n",
       "              1298,\n",
       "              1300,\n",
       "              1302,\n",
       "              1303,\n",
       "              1304,\n",
       "              1305,\n",
       "              1306,\n",
       "              1307,\n",
       "              1308,\n",
       "              1309,\n",
       "              1310,\n",
       "              1311,\n",
       "              1312,\n",
       "              1315,\n",
       "              1318,\n",
       "              1319,\n",
       "              1320,\n",
       "              1321,\n",
       "              1324,\n",
       "              1325,\n",
       "              1326,\n",
       "              1328,\n",
       "              1330,\n",
       "              1331,\n",
       "              1332,\n",
       "              1333,\n",
       "              1336,\n",
       "              1337,\n",
       "              1338,\n",
       "              1339,\n",
       "              1341,\n",
       "              1344,\n",
       "              1348,\n",
       "              1349,\n",
       "              1352,\n",
       "              1354,\n",
       "              1355,\n",
       "              1357,\n",
       "              1358,\n",
       "              1361,\n",
       "              1362,\n",
       "              1364,\n",
       "              1365,\n",
       "              1366,\n",
       "              1367,\n",
       "              1369,\n",
       "              1371,\n",
       "              1373,\n",
       "              1374,\n",
       "              1376,\n",
       "              1377,\n",
       "              1378,\n",
       "              1379,\n",
       "              1381,\n",
       "              1382,\n",
       "              1384,\n",
       "              1385,\n",
       "              1386,\n",
       "              1388,\n",
       "              1389,\n",
       "              1391,\n",
       "              1392,\n",
       "              1393,\n",
       "              1395,\n",
       "              1396,\n",
       "              1398,\n",
       "              1399,\n",
       "              1400,\n",
       "              1401,\n",
       "              1402,\n",
       "              1403,\n",
       "              1404,\n",
       "              1405,\n",
       "              1406,\n",
       "              1408,\n",
       "              1409,\n",
       "              1411,\n",
       "              1412,\n",
       "              1413,\n",
       "              1414,\n",
       "              1418,\n",
       "              1419,\n",
       "              1420,\n",
       "              1421,\n",
       "              1422,\n",
       "              1423,\n",
       "              1424,\n",
       "              1425,\n",
       "              ...},\n",
       "             'male': {0,\n",
       "              5,\n",
       "              7,\n",
       "              9,\n",
       "              11,\n",
       "              14,\n",
       "              15,\n",
       "              16,\n",
       "              18,\n",
       "              19,\n",
       "              23,\n",
       "              30,\n",
       "              35,\n",
       "              37,\n",
       "              39,\n",
       "              42,\n",
       "              44,\n",
       "              45,\n",
       "              48,\n",
       "              49,\n",
       "              51,\n",
       "              55,\n",
       "              56,\n",
       "              58,\n",
       "              59,\n",
       "              62,\n",
       "              73,\n",
       "              75,\n",
       "              76,\n",
       "              78,\n",
       "              81,\n",
       "              83,\n",
       "              84,\n",
       "              86,\n",
       "              94,\n",
       "              98,\n",
       "              100,\n",
       "              103,\n",
       "              106,\n",
       "              110,\n",
       "              116,\n",
       "              128,\n",
       "              133,\n",
       "              142,\n",
       "              144,\n",
       "              146,\n",
       "              153,\n",
       "              156,\n",
       "              158,\n",
       "              169,\n",
       "              175,\n",
       "              188,\n",
       "              189,\n",
       "              195,\n",
       "              196,\n",
       "              203,\n",
       "              205,\n",
       "              206,\n",
       "              207,\n",
       "              209,\n",
       "              225,\n",
       "              233,\n",
       "              239,\n",
       "              242,\n",
       "              249,\n",
       "              251,\n",
       "              253,\n",
       "              259,\n",
       "              260,\n",
       "              261,\n",
       "              263,\n",
       "              265,\n",
       "              266,\n",
       "              268,\n",
       "              269,\n",
       "              272,\n",
       "              275,\n",
       "              276,\n",
       "              286,\n",
       "              292,\n",
       "              294,\n",
       "              296,\n",
       "              300,\n",
       "              301,\n",
       "              307,\n",
       "              308,\n",
       "              311,\n",
       "              316,\n",
       "              318,\n",
       "              319,\n",
       "              323,\n",
       "              327,\n",
       "              328,\n",
       "              329,\n",
       "              330,\n",
       "              331,\n",
       "              332,\n",
       "              335,\n",
       "              336,\n",
       "              344,\n",
       "              345,\n",
       "              352,\n",
       "              354,\n",
       "              357,\n",
       "              360,\n",
       "              369,\n",
       "              371,\n",
       "              373,\n",
       "              374,\n",
       "              375,\n",
       "              376,\n",
       "              382,\n",
       "              384,\n",
       "              386,\n",
       "              387,\n",
       "              390,\n",
       "              391,\n",
       "              393,\n",
       "              396,\n",
       "              397,\n",
       "              400,\n",
       "              403,\n",
       "              411,\n",
       "              417,\n",
       "              422,\n",
       "              426,\n",
       "              430,\n",
       "              431,\n",
       "              437,\n",
       "              440,\n",
       "              441,\n",
       "              442,\n",
       "              444,\n",
       "              448,\n",
       "              451,\n",
       "              452,\n",
       "              461,\n",
       "              462,\n",
       "              467,\n",
       "              469,\n",
       "              471,\n",
       "              475,\n",
       "              477,\n",
       "              481,\n",
       "              488,\n",
       "              490,\n",
       "              496,\n",
       "              497,\n",
       "              498,\n",
       "              499,\n",
       "              512,\n",
       "              513,\n",
       "              517,\n",
       "              521,\n",
       "              522,\n",
       "              523,\n",
       "              524,\n",
       "              525,\n",
       "              526,\n",
       "              528,\n",
       "              533,\n",
       "              534,\n",
       "              542,\n",
       "              544,\n",
       "              554,\n",
       "              562,\n",
       "              563,\n",
       "              564,\n",
       "              565,\n",
       "              568,\n",
       "              569,\n",
       "              572,\n",
       "              574,\n",
       "              575,\n",
       "              576,\n",
       "              579,\n",
       "              580,\n",
       "              582,\n",
       "              583,\n",
       "              585,\n",
       "              587,\n",
       "              588,\n",
       "              597,\n",
       "              599,\n",
       "              600,\n",
       "              602,\n",
       "              609,\n",
       "              610,\n",
       "              611,\n",
       "              624,\n",
       "              633,\n",
       "              641,\n",
       "              642,\n",
       "              646,\n",
       "              650,\n",
       "              651,\n",
       "              657,\n",
       "              663,\n",
       "              665,\n",
       "              675,\n",
       "              679,\n",
       "              682,\n",
       "              683,\n",
       "              689,\n",
       "              698,\n",
       "              699,\n",
       "              700,\n",
       "              705,\n",
       "              706,\n",
       "              709,\n",
       "              713,\n",
       "              716,\n",
       "              718,\n",
       "              719,\n",
       "              729,\n",
       "              732,\n",
       "              735,\n",
       "              737,\n",
       "              738,\n",
       "              740,\n",
       "              744,\n",
       "              750,\n",
       "              751,\n",
       "              762,\n",
       "              765,\n",
       "              767,\n",
       "              774,\n",
       "              781,\n",
       "              783,\n",
       "              788,\n",
       "              789,\n",
       "              790,\n",
       "              791,\n",
       "              793,\n",
       "              796,\n",
       "              797,\n",
       "              798,\n",
       "              802,\n",
       "              803,\n",
       "              804,\n",
       "              806,\n",
       "              812,\n",
       "              814,\n",
       "              819,\n",
       "              821,\n",
       "              825,\n",
       "              826,\n",
       "              830,\n",
       "              833,\n",
       "              834,\n",
       "              835,\n",
       "              836,\n",
       "              837,\n",
       "              840,\n",
       "              841,\n",
       "              843,\n",
       "              850,\n",
       "              852,\n",
       "              854,\n",
       "              863,\n",
       "              868,\n",
       "              870,\n",
       "              875,\n",
       "              876,\n",
       "              877,\n",
       "              879,\n",
       "              880,\n",
       "              881,\n",
       "              882,\n",
       "              885,\n",
       "              889,\n",
       "              894,\n",
       "              895,\n",
       "              900,\n",
       "              903,\n",
       "              907,\n",
       "              913,\n",
       "              924,\n",
       "              929,\n",
       "              930,\n",
       "              933,\n",
       "              937,\n",
       "              938,\n",
       "              944,\n",
       "              954,\n",
       "              958,\n",
       "              962,\n",
       "              964,\n",
       "              965,\n",
       "              968,\n",
       "              970,\n",
       "              971,\n",
       "              975,\n",
       "              980,\n",
       "              981,\n",
       "              982,\n",
       "              984,\n",
       "              986,\n",
       "              989,\n",
       "              991,\n",
       "              992,\n",
       "              994,\n",
       "              998,\n",
       "              1002,\n",
       "              1007,\n",
       "              1012,\n",
       "              1015,\n",
       "              1018,\n",
       "              1019,\n",
       "              1020,\n",
       "              1027,\n",
       "              1029,\n",
       "              1032,\n",
       "              1033,\n",
       "              1041,\n",
       "              1043,\n",
       "              1045,\n",
       "              1050,\n",
       "              1052,\n",
       "              1053,\n",
       "              1058,\n",
       "              1061,\n",
       "              1085,\n",
       "              1088,\n",
       "              1092,\n",
       "              1097,\n",
       "              1099,\n",
       "              1100,\n",
       "              1103,\n",
       "              1106,\n",
       "              1109,\n",
       "              1112,\n",
       "              1114,\n",
       "              1115,\n",
       "              1119,\n",
       "              1120,\n",
       "              1123,\n",
       "              1124,\n",
       "              1129,\n",
       "              1134,\n",
       "              1135,\n",
       "              1145,\n",
       "              1146,\n",
       "              1149,\n",
       "              1150,\n",
       "              1155,\n",
       "              1156,\n",
       "              1159,\n",
       "              1163,\n",
       "              1166,\n",
       "              1168,\n",
       "              1172,\n",
       "              1173,\n",
       "              1179,\n",
       "              1183,\n",
       "              1187,\n",
       "              1188,\n",
       "              1192,\n",
       "              1199,\n",
       "              1200,\n",
       "              1203,\n",
       "              1206,\n",
       "              1207,\n",
       "              1209,\n",
       "              1210,\n",
       "              1213,\n",
       "              1217,\n",
       "              1218,\n",
       "              1224,\n",
       "              1225,\n",
       "              1231,\n",
       "              1232,\n",
       "              1236,\n",
       "              1237,\n",
       "              1240,\n",
       "              1243,\n",
       "              1246,\n",
       "              1249,\n",
       "              1252,\n",
       "              1260,\n",
       "              1262,\n",
       "              1264,\n",
       "              1277,\n",
       "              1278,\n",
       "              1287,\n",
       "              1292,\n",
       "              1299,\n",
       "              1301,\n",
       "              1313,\n",
       "              1314,\n",
       "              1316,\n",
       "              1317,\n",
       "              1322,\n",
       "              1323,\n",
       "              1327,\n",
       "              1329,\n",
       "              1334,\n",
       "              1335,\n",
       "              1340,\n",
       "              1342,\n",
       "              1343,\n",
       "              1345,\n",
       "              1346,\n",
       "              1347,\n",
       "              1350,\n",
       "              1351,\n",
       "              1353,\n",
       "              1356,\n",
       "              1359,\n",
       "              1360,\n",
       "              1363,\n",
       "              1368,\n",
       "              1370,\n",
       "              1372,\n",
       "              1375,\n",
       "              1380,\n",
       "              1383,\n",
       "              1387,\n",
       "              1390,\n",
       "              1394,\n",
       "              1397,\n",
       "              1407,\n",
       "              1410,\n",
       "              1415,\n",
       "              1416,\n",
       "              1417,\n",
       "              1429,\n",
       "              1430,\n",
       "              1437,\n",
       "              1440,\n",
       "              1441,\n",
       "              1447,\n",
       "              1448,\n",
       "              1451,\n",
       "              1452,\n",
       "              1459,\n",
       "              1461,\n",
       "              1464,\n",
       "              1465,\n",
       "              1466,\n",
       "              1467,\n",
       "              1469,\n",
       "              1475,\n",
       "              1476,\n",
       "              1477,\n",
       "              1491,\n",
       "              1497,\n",
       "              1501,\n",
       "              1503,\n",
       "              1504,\n",
       "              1505,\n",
       "              1506,\n",
       "              1511,\n",
       "              1514,\n",
       "              1519,\n",
       "              1520,\n",
       "              1523,\n",
       "              1524,\n",
       "              1525,\n",
       "              1530,\n",
       "              1531,\n",
       "              1532,\n",
       "              1536,\n",
       "              1538,\n",
       "              1539,\n",
       "              1547,\n",
       "              1552,\n",
       "              1554,\n",
       "              1555,\n",
       "              1557,\n",
       "              1560,\n",
       "              1561,\n",
       "              1564,\n",
       "              1568,\n",
       "              1572,\n",
       "              1575,\n",
       "              1584}})"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "testsets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "CLASSIFIER 1\n",
      "------------ \n",
      "Female precision: 0.78\n",
      "Male precision: 0.73\n",
      "Female recall: 0.87\n",
      "Male recall: 0.58\n",
      "Female F1 score: 0.82\n",
      "Male F1 score: 0.65\n",
      "\n",
      "CLASSIFIER 2\n",
      "------------ \n",
      "Female precision: 0.82\n",
      "Male precision: 0.71\n",
      "Female recall: 0.83\n",
      "Male recall: 0.69\n",
      "Female F1 score: 0.83\n",
      "Male F1 score: 0.7\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from nltk.metrics.scores import (precision, recall, f_measure)\n",
    "\n",
    "# We can proceed to print the metrics for each classifier. However, we cannot get the accuracy in this manner because nltk.metrics.scores.accuracy(reference, test) works by comparing test[i] == reference[i] and our reference and test are not formatted in a way that allows for this. It's the same for the confusion matrix.\n",
    "args = (\n",
    "    round(precision(refsets['female'], testsets['female']), 2),\n",
    "    round(precision(refsets['male'], testsets['male']), 2),\n",
    "    round(recall(refsets['female'], testsets['female']), 2),\n",
    "    round(recall(refsets['male'], testsets['male']), 2),\n",
    "    round(f_measure(refsets['female'], testsets['female']), 2),\n",
    "    round(f_measure(refsets['male'], testsets['male']), 2)\n",
    ")\n",
    "\n",
    "args2 = (\n",
    "    round(precision(refsets2['female'], testsets2['female']), 2),\n",
    "    round(precision(refsets2['male'], testsets2['male']), 2),\n",
    "    round(recall(refsets2['female'], testsets2['female']), 2),\n",
    "    round(recall(refsets2['male'], testsets2['male']), 2),\n",
    "    round(f_measure(refsets2['female'], testsets2['female']), 2),\n",
    "    round(f_measure(refsets2['male'], testsets2['male']), 2)\n",
    ")\n",
    "\n",
    "print('''\n",
    "CLASSIFIER 1\n",
    "------------ \n",
    "Female precision: {0}\n",
    "Male precision: {1}\n",
    "Female recall: {2}\n",
    "Male recall: {3}\n",
    "Female F1 score: {4}\n",
    "Male F1 score: {5}\n",
    "\n",
    "CLASSIFIER 2\n",
    "------------ \n",
    "Female precision: {6}\n",
    "Male precision: {7}\n",
    "Female recall: {8}\n",
    "Male recall: {9}\n",
    "Female F1 score: {10}\n",
    "Male F1 score: {11}\n",
    "'''.format(*args, *args2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Error analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('male', 'female', 'Hersh'),\n",
       " ('female', 'male', 'Ros'),\n",
       " ('female', 'male', 'Kerrin'),\n",
       " ('female', 'male', 'Shaylynn'),\n",
       " ('male', 'female', 'Tully')]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "errors = []\n",
    "for (name, tag) in test_names:\n",
    "    guess = classifier2.classify(gender_features(name))\n",
    "    if guess != tag:\n",
    "        errors.append((tag, guess, name))\n",
    "\n",
    "errors[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Correct = female   guess = male     name = Adriaens\n",
      "Correct = female   guess = male     name = Alis\n",
      "Correct = female   guess = male     name = Allsun\n",
      "Correct = female   guess = male     name = Alys\n",
      "Correct = female   guess = male     name = Anet\n",
      "Correct = female   guess = male     name = Ann\n",
      "Correct = female   guess = male     name = Ashlen\n",
      "Correct = female   guess = male     name = Barb\n",
      "Correct = female   guess = male     name = Beitris\n",
      "Correct = female   guess = male     name = Berget\n",
      "Correct = female   guess = male     name = Bev\n",
      "Correct = female   guess = male     name = Bidget\n",
      "Correct = female   guess = male     name = Brenn\n",
      "Correct = female   guess = male     name = Brett\n",
      "Correct = female   guess = male     name = Brigit\n",
      "Correct = female   guess = male     name = Brynn\n",
      "Correct = female   guess = male     name = Carlyn\n",
      "Correct = female   guess = male     name = Carmen\n",
      "Correct = female   guess = male     name = Caro\n",
      "Correct = female   guess = male     name = Cass\n",
      "Correct = female   guess = male     name = Charlott\n",
      "Correct = female   guess = male     name = Clem\n",
      "Correct = female   guess = male     name = Con\n",
      "Correct = female   guess = male     name = Cristen\n",
      "Correct = female   guess = male     name = Cristin\n",
      "Correct = female   guess = male     name = Delores\n",
      "Correct = female   guess = male     name = Demeter\n",
      "Correct = female   guess = male     name = Devin\n",
      "Correct = female   guess = male     name = Donnajean\n",
      "Correct = female   guess = male     name = Eleanor\n",
      "Correct = female   guess = male     name = Elinor\n",
      "Correct = female   guess = male     name = Eryn\n",
      "Correct = female   guess = male     name = Farand\n",
      "Correct = female   guess = male     name = Felicdad\n",
      "Correct = female   guess = male     name = Fleur\n",
      "Correct = female   guess = male     name = Flor\n",
      "Correct = female   guess = male     name = Floris\n",
      "Correct = female   guess = male     name = Gaynor\n",
      "Correct = female   guess = male     name = Gertrudis\n",
      "Correct = female   guess = male     name = Gilligan\n",
      "Correct = female   guess = male     name = Glad\n",
      "Correct = female   guess = male     name = Gladys\n",
      "Correct = female   guess = male     name = Grier\n",
      "Correct = female   guess = male     name = Gwendolin\n",
      "Correct = female   guess = male     name = Hedwig\n",
      "Correct = female   guess = male     name = Hildegaard\n",
      "Correct = female   guess = male     name = Honor\n",
      "Correct = female   guess = male     name = Ingeberg\n",
      "Correct = female   guess = male     name = Inger\n",
      "Correct = female   guess = male     name = Isador\n",
      "Correct = female   guess = male     name = Jacquelynn\n",
      "Correct = female   guess = male     name = Jaquelin\n",
      "Correct = female   guess = male     name = Jasmin\n",
      "Correct = female   guess = male     name = Jenn\n",
      "Correct = female   guess = male     name = Jess\n",
      "Correct = female   guess = male     name = Jo-Ann\n",
      "Correct = female   guess = male     name = Joann\n",
      "Correct = female   guess = male     name = Joannes\n",
      "Correct = female   guess = male     name = Joellen\n",
      "Correct = female   guess = male     name = Joellyn\n",
      "Correct = female   guess = male     name = Joyan\n",
      "Correct = female   guess = male     name = Juliann\n",
      "Correct = female   guess = male     name = Juliet\n",
      "Correct = female   guess = male     name = Justin\n",
      "Correct = female   guess = male     name = Karilynn\n",
      "Correct = female   guess = male     name = Katleen\n",
      "Correct = female   guess = male     name = Kellen\n",
      "Correct = female   guess = male     name = Keriann\n",
      "Correct = female   guess = male     name = Kerrin\n",
      "Correct = female   guess = male     name = Kirsten\n",
      "Correct = female   guess = male     name = Kit\n",
      "Correct = female   guess = male     name = Koren\n",
      "Correct = female   guess = male     name = Kristyn\n",
      "Correct = female   guess = male     name = Lamb\n",
      "Correct = female   guess = male     name = Lauryn\n",
      "Correct = female   guess = male     name = Lib\n",
      "Correct = female   guess = male     name = Lillian\n",
      "Correct = female   guess = male     name = Lillis\n",
      "Correct = female   guess = male     name = Linet\n",
      "Correct = female   guess = male     name = Linnet\n",
      "Correct = female   guess = male     name = Lois\n",
      "Correct = female   guess = male     name = Lorrin\n",
      "Correct = female   guess = male     name = Madlin\n",
      "Correct = female   guess = male     name = Margalo\n",
      "Correct = female   guess = male     name = Margo\n",
      "Correct = female   guess = male     name = Mariellen\n",
      "Correct = female   guess = male     name = Marijo\n",
      "Correct = female   guess = male     name = Maris\n",
      "Correct = female   guess = male     name = Marlo\n",
      "Correct = female   guess = male     name = Maryann\n",
      "Correct = female   guess = male     name = Mavis\n",
      "Correct = female   guess = male     name = Megan\n",
      "Correct = female   guess = male     name = Meghann\n",
      "Correct = female   guess = male     name = Millisent\n",
      "Correct = female   guess = male     name = Morgen\n",
      "Correct = female   guess = male     name = Muffin\n",
      "Correct = female   guess = male     name = Nan\n",
      "Correct = female   guess = male     name = Nariko\n",
      "Correct = female   guess = male     name = Ninon\n",
      "Correct = female   guess = male     name = Noellyn\n",
      "Correct = female   guess = male     name = Noreen\n",
      "Correct = female   guess = male     name = Olwen\n",
      "Correct = female   guess = male     name = Pam\n",
      "Correct = female   guess = male     name = Pat\n",
      "Correct = female   guess = male     name = Phyllys\n",
      "Correct = female   guess = male     name = Quinn\n",
      "Correct = female   guess = male     name = Robbyn\n",
      "Correct = female   guess = male     name = Robinet\n",
      "Correct = female   guess = male     name = Robyn\n",
      "Correct = female   guess = male     name = Ros\n",
      "Correct = female   guess = male     name = Roseann\n",
      "Correct = female   guess = male     name = Roslyn\n",
      "Correct = female   guess = male     name = Row\n",
      "Correct = female   guess = male     name = Rozalin\n",
      "Correct = female   guess = male     name = Sam\n",
      "Correct = female   guess = male     name = Sara-Ann\n",
      "Correct = female   guess = male     name = Shamit\n",
      "Correct = female   guess = male     name = Sharleen\n",
      "Correct = female   guess = male     name = Shaylynn\n",
      "Correct = female   guess = male     name = Sheila-Kathryn\n",
      "Correct = female   guess = male     name = Sherilyn\n",
      "Correct = female   guess = male     name = Shir\n",
      "Correct = female   guess = male     name = Shirleen\n",
      "Correct = female   guess = male     name = Sigrid\n",
      "Correct = female   guess = male     name = Storm\n",
      "Correct = female   guess = male     name = Suzann\n",
      "Correct = female   guess = male     name = Vin\n",
      "Correct = female   guess = male     name = Vivyan\n",
      "Correct = female   guess = male     name = Willyt\n",
      "Correct = female   guess = male     name = Yoshiko\n",
      "Correct = male     guess = female   name = Abbey\n",
      "Correct = male     guess = female   name = Abel\n",
      "Correct = male     guess = female   name = Aguste\n",
      "Correct = male     guess = female   name = Alley\n",
      "Correct = male     guess = female   name = Alphonse\n",
      "Correct = male     guess = female   name = Ambrosi\n",
      "Correct = male     guess = female   name = Amery\n",
      "Correct = male     guess = female   name = Andrej\n",
      "Correct = male     guess = female   name = Andrzej\n",
      "Correct = male     guess = female   name = Antone\n",
      "Correct = male     guess = female   name = Arel\n",
      "Correct = male     guess = female   name = Arie\n",
      "Correct = male     guess = female   name = Aristotle\n",
      "Correct = male     guess = female   name = Arvie\n",
      "Correct = male     guess = female   name = Ash\n",
      "Correct = male     guess = female   name = Averell\n",
      "Correct = male     guess = female   name = Baillie\n",
      "Correct = male     guess = female   name = Barth\n",
      "Correct = male     guess = female   name = Bartholemy\n",
      "Correct = male     guess = female   name = Barty\n",
      "Correct = male     guess = female   name = Basil\n",
      "Correct = male     guess = female   name = Bay\n",
      "Correct = male     guess = female   name = Bela\n",
      "Correct = male     guess = female   name = Benji\n",
      "Correct = male     guess = female   name = Bentley\n",
      "Correct = male     guess = female   name = Bernie\n",
      "Correct = male     guess = female   name = Billy\n",
      "Correct = male     guess = female   name = Boniface\n",
      "Correct = male     guess = female   name = Boyce\n",
      "Correct = male     guess = female   name = Brody\n",
      "Correct = male     guess = female   name = Bucky\n",
      "Correct = male     guess = female   name = Cal\n",
      "Correct = male     guess = female   name = Casey\n",
      "Correct = male     guess = female   name = Chancey\n",
      "Correct = male     guess = female   name = Chaunce\n",
      "Correct = male     guess = female   name = Che\n",
      "Correct = male     guess = female   name = Chrisy\n",
      "Correct = male     guess = female   name = Corby\n",
      "Correct = male     guess = female   name = Cortese\n",
      "Correct = male     guess = female   name = Courtney\n",
      "Correct = male     guess = female   name = Curtice\n",
      "Correct = male     guess = female   name = Cyrill\n",
      "Correct = male     guess = female   name = Cyrille\n",
      "Correct = male     guess = female   name = Danie\n",
      "Correct = male     guess = female   name = Dannie\n",
      "Correct = male     guess = female   name = Darcy\n",
      "Correct = male     guess = female   name = Darrel\n",
      "Correct = male     guess = female   name = Darrell\n",
      "Correct = male     guess = female   name = Davidde\n",
      "Correct = male     guess = female   name = Denny\n",
      "Correct = male     guess = female   name = Derby\n",
      "Correct = male     guess = female   name = Dickie\n",
      "Correct = male     guess = female   name = Dimitry\n",
      "Correct = male     guess = female   name = Dominique\n",
      "Correct = male     guess = female   name = Douggie\n",
      "Correct = male     guess = female   name = Duane\n",
      "Correct = male     guess = female   name = Duffy\n",
      "Correct = male     guess = female   name = Dwaine\n",
      "Correct = male     guess = female   name = Earl\n",
      "Correct = male     guess = female   name = Earle\n",
      "Correct = male     guess = female   name = Edie\n",
      "Correct = male     guess = female   name = Elijah\n",
      "Correct = male     guess = female   name = Ellsworth\n",
      "Correct = male     guess = female   name = Elroy\n",
      "Correct = male     guess = female   name = Emanuel\n",
      "Correct = male     guess = female   name = Emmery\n",
      "Correct = male     guess = female   name = Enrique\n",
      "Correct = male     guess = female   name = Erich\n",
      "Correct = male     guess = female   name = Etienne\n",
      "Correct = male     guess = female   name = Ezekiel\n",
      "Correct = male     guess = female   name = Ferdy\n",
      "Correct = male     guess = female   name = Fidel\n",
      "Correct = male     guess = female   name = Freddie\n",
      "Correct = male     guess = female   name = Frederich\n",
      "Correct = male     guess = female   name = Gamaliel\n",
      "Correct = male     guess = female   name = Garcia\n",
      "Correct = male     guess = female   name = Garey\n",
      "Correct = male     guess = female   name = Garth\n",
      "Correct = male     guess = female   name = Gary\n",
      "Correct = male     guess = female   name = Gay\n",
      "Correct = male     guess = female   name = Gayle\n",
      "Correct = male     guess = female   name = Gene\n",
      "Correct = male     guess = female   name = Gere\n",
      "Correct = male     guess = female   name = Geri\n",
      "Correct = male     guess = female   name = Germaine\n",
      "Correct = male     guess = female   name = Gerry\n",
      "Correct = male     guess = female   name = Giavani\n",
      "Correct = male     guess = female   name = Giuseppe\n",
      "Correct = male     guess = female   name = Goose\n",
      "Correct = male     guess = female   name = Gregory\n",
      "Correct = male     guess = female   name = Grove\n",
      "Correct = male     guess = female   name = Hari\n",
      "Correct = male     guess = female   name = Hartley\n",
      "Correct = male     guess = female   name = Harwell\n",
      "Correct = male     guess = female   name = Haskel\n",
      "Correct = male     guess = female   name = Heath\n",
      "Correct = male     guess = female   name = Heinrich\n",
      "Correct = male     guess = female   name = Henri\n",
      "Correct = male     guess = female   name = Hersh\n",
      "Correct = male     guess = female   name = Hewe\n",
      "Correct = male     guess = female   name = Hewie\n",
      "Correct = male     guess = female   name = Hezekiah\n",
      "Correct = male     guess = female   name = Hilary\n",
      "Correct = male     guess = female   name = Hillary\n",
      "Correct = male     guess = female   name = Hiralal\n",
      "Correct = male     guess = female   name = Hirsch\n",
      "Correct = male     guess = female   name = Hodge\n",
      "Correct = male     guess = female   name = Hy\n",
      "Correct = male     guess = female   name = Iggie\n",
      "Correct = male     guess = female   name = Iggy\n",
      "Correct = male     guess = female   name = Israel\n",
      "Correct = male     guess = female   name = Jake\n",
      "Correct = male     guess = female   name = Jean-Paul\n",
      "Correct = male     guess = female   name = Jeffrey\n",
      "Correct = male     guess = female   name = Jerome\n",
      "Correct = male     guess = female   name = Jeromy\n",
      "Correct = male     guess = female   name = Jerrie\n",
      "Correct = male     guess = female   name = Jerrome\n",
      "Correct = male     guess = female   name = Jimmy\n",
      "Correct = male     guess = female   name = Jodie\n",
      "Correct = male     guess = female   name = Joe\n",
      "Correct = male     guess = female   name = Josiah\n",
      "Correct = male     guess = female   name = Jule\n",
      "Correct = male     guess = female   name = Keene\n",
      "Correct = male     guess = female   name = Kelly\n",
      "Correct = male     guess = female   name = Kennedy\n",
      "Correct = male     guess = female   name = Kimball\n",
      "Correct = male     guess = female   name = Kingsley\n",
      "Correct = male     guess = female   name = Knox\n",
      "Correct = male     guess = female   name = Lanny\n",
      "Correct = male     guess = female   name = Laurance\n",
      "Correct = male     guess = female   name = Laurence\n",
      "Correct = male     guess = female   name = Lemuel\n",
      "Correct = male     guess = female   name = Levy\n",
      "Correct = male     guess = female   name = Lowell\n",
      "Correct = male     guess = female   name = Mace\n",
      "Correct = male     guess = female   name = Mahesh\n",
      "Correct = male     guess = female   name = Marcel\n",
      "Correct = male     guess = female   name = Marlowe\n",
      "Correct = male     guess = female   name = Marty\n",
      "Correct = male     guess = female   name = Mattie\n",
      "Correct = male     guess = female   name = Maurice\n",
      "Correct = male     guess = female   name = Merrel\n",
      "Correct = male     guess = female   name = Meryl\n",
      "Correct = male     guess = female   name = Micah\n",
      "Correct = male     guess = female   name = Michal\n",
      "Correct = male     guess = female   name = Micheal\n",
      "Correct = male     guess = female   name = Michel\n",
      "Correct = male     guess = female   name = Michele\n",
      "Correct = male     guess = female   name = Micky\n",
      "Correct = male     guess = female   name = Miguel\n",
      "Correct = male     guess = female   name = Mikael\n",
      "Correct = male     guess = female   name = Mikey\n",
      "Correct = male     guess = female   name = Mitchel\n",
      "Correct = male     guess = female   name = Montague\n",
      "Correct = male     guess = female   name = Morrie\n",
      "Correct = male     guess = female   name = Morty\n",
      "Correct = male     guess = female   name = Mustafa\n",
      "Correct = male     guess = female   name = Nate\n",
      "Correct = male     guess = female   name = Nathanael\n",
      "Correct = male     guess = female   name = Nathanial\n",
      "Correct = male     guess = female   name = Nicky\n",
      "Correct = male     guess = female   name = Noah\n",
      "Correct = male     guess = female   name = Normie\n",
      "Correct = male     guess = female   name = Oral\n",
      "Correct = male     guess = female   name = Orbadiah\n",
      "Correct = male     guess = female   name = Osborne\n",
      "Correct = male     guess = female   name = Ozzie\n",
      "Correct = male     guess = female   name = Partha\n",
      "Correct = male     guess = female   name = Pascal\n",
      "Correct = male     guess = female   name = Pavel\n",
      "Correct = male     guess = female   name = Petey\n",
      "Correct = male     guess = female   name = Piggy\n",
      "Correct = male     guess = female   name = Pooh\n",
      "Correct = male     guess = female   name = Powell\n",
      "Correct = male     guess = female   name = Pryce\n",
      "Correct = male     guess = female   name = Purcell\n",
      "Correct = male     guess = female   name = Quill\n",
      "Correct = male     guess = female   name = Rafael\n",
      "Correct = male     guess = female   name = Rafe\n",
      "Correct = male     guess = female   name = Ralph\n",
      "Correct = male     guess = female   name = Randal\n",
      "Correct = male     guess = female   name = Ravi\n",
      "Correct = male     guess = female   name = Reggy\n",
      "Correct = male     guess = female   name = Riley\n",
      "Correct = male     guess = female   name = Ronnie\n",
      "Correct = male     guess = female   name = Ronny\n",
      "Correct = male     guess = female   name = Roscoe\n",
      "Correct = male     guess = female   name = Ruddie\n",
      "Correct = male     guess = female   name = Russel\n",
      "Correct = male     guess = female   name = Russell\n",
      "Correct = male     guess = female   name = Salvatore\n",
      "Correct = male     guess = female   name = Sawyere\n",
      "Correct = male     guess = female   name = Sax\n",
      "Correct = male     guess = female   name = Scottie\n",
      "Correct = male     guess = female   name = Scotty\n",
      "Correct = male     guess = female   name = Selby\n",
      "Correct = male     guess = female   name = Serge\n",
      "Correct = male     guess = female   name = Shane\n",
      "Correct = male     guess = female   name = Si\n",
      "Correct = male     guess = female   name = Skippy\n",
      "Correct = male     guess = female   name = Sloane\n",
      "Correct = male     guess = female   name = Sly\n",
      "Correct = male     guess = female   name = Smith\n",
      "Correct = male     guess = female   name = Stanleigh\n",
      "Correct = male     guess = female   name = Stanly\n",
      "Correct = male     guess = female   name = Sterne\n",
      "Correct = male     guess = female   name = Tabbie\n",
      "Correct = male     guess = female   name = Tabby\n",
      "Correct = male     guess = female   name = Tallie\n",
      "Correct = male     guess = female   name = Thayne\n",
      "Correct = male     guess = female   name = Tobe\n",
      "Correct = male     guess = female   name = Toby\n",
      "Correct = male     guess = female   name = Toddie\n",
      "Correct = male     guess = female   name = Toddy\n",
      "Correct = male     guess = female   name = Tome\n",
      "Correct = male     guess = female   name = Tommie\n",
      "Correct = male     guess = female   name = Torey\n",
      "Correct = male     guess = female   name = Torre\n",
      "Correct = male     guess = female   name = Torrey\n",
      "Correct = male     guess = female   name = Torry\n",
      "Correct = male     guess = female   name = Towny\n",
      "Correct = male     guess = female   name = Tray\n",
      "Correct = male     guess = female   name = Tre\n",
      "Correct = male     guess = female   name = Tuckie\n",
      "Correct = male     guess = female   name = Tucky\n",
      "Correct = male     guess = female   name = Tully\n",
      "Correct = male     guess = female   name = Tyrone\n",
      "Correct = male     guess = female   name = Uli\n",
      "Correct = male     guess = female   name = Uri\n",
      "Correct = male     guess = female   name = Uriel\n",
      "Correct = male     guess = female   name = Vassili\n",
      "Correct = male     guess = female   name = Verne\n",
      "Correct = male     guess = female   name = Vince\n",
      "Correct = male     guess = female   name = Voltaire\n",
      "Correct = male     guess = female   name = Wake\n",
      "Correct = male     guess = female   name = Walsh\n",
      "Correct = male     guess = female   name = Welsh\n",
      "Correct = male     guess = female   name = Weslie\n",
      "Correct = male     guess = female   name = Willi\n",
      "Correct = male     guess = female   name = Willie\n",
      "Correct = male     guess = female   name = Wojciech\n",
      "Correct = male     guess = female   name = Woody\n",
      "Correct = male     guess = female   name = Wylie\n",
      "Correct = male     guess = female   name = Yardley\n",
      "Correct = male     guess = female   name = Zacharia\n",
      "Correct = male     guess = female   name = Zacharie\n",
      "Correct = male     guess = female   name = Zacherie\n",
      "Correct = male     guess = female   name = Zollie\n"
     ]
    }
   ],
   "source": [
    "for (tag, guess, name) in sorted(errors):\n",
    "    print('Correct = {:8} guess = {:8} name = {}'.format(tag, guess, name)) # :8 creates spaces between columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking through this list of errors, it seems that some suffixes that are more than one letter long can be indicative of name genders. For example, names ending in *yn* appear to be predominantly female, despite the fact that names ending in *n* tend to be male; and names ending in *ch* are usually male, even though names that end in *h* tend to be female."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Building a classifier with even more features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'count(a)': 0,\n",
       " 'count(b)': 0,\n",
       " 'count(c)': 0,\n",
       " 'count(d)': 0,\n",
       " 'count(e)': 0,\n",
       " 'count(f)': 0,\n",
       " 'count(g)': 0,\n",
       " 'count(h)': 1,\n",
       " 'count(i)': 0,\n",
       " 'count(j)': 1,\n",
       " 'count(k)': 0,\n",
       " 'count(l)': 0,\n",
       " 'count(m)': 0,\n",
       " 'count(n)': 1,\n",
       " 'count(o)': 1,\n",
       " 'count(p)': 0,\n",
       " 'count(q)': 0,\n",
       " 'count(r)': 0,\n",
       " 'count(s)': 0,\n",
       " 'count(t)': 0,\n",
       " 'count(u)': 0,\n",
       " 'count(v)': 0,\n",
       " 'count(w)': 0,\n",
       " 'count(x)': 0,\n",
       " 'count(y)': 0,\n",
       " 'count(z)': 0,\n",
       " 'first_letter': 'j',\n",
       " 'has(a)': False,\n",
       " 'has(b)': False,\n",
       " 'has(c)': False,\n",
       " 'has(d)': False,\n",
       " 'has(e)': False,\n",
       " 'has(f)': False,\n",
       " 'has(g)': False,\n",
       " 'has(h)': True,\n",
       " 'has(i)': False,\n",
       " 'has(j)': True,\n",
       " 'has(k)': False,\n",
       " 'has(l)': False,\n",
       " 'has(m)': False,\n",
       " 'has(n)': True,\n",
       " 'has(o)': True,\n",
       " 'has(p)': False,\n",
       " 'has(q)': False,\n",
       " 'has(r)': False,\n",
       " 'has(s)': False,\n",
       " 'has(t)': False,\n",
       " 'has(u)': False,\n",
       " 'has(v)': False,\n",
       " 'has(w)': False,\n",
       " 'has(x)': False,\n",
       " 'has(y)': False,\n",
       " 'has(z)': False,\n",
       " 'suffix1': 'n',\n",
       " 'suffix2': 'hn'}"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def gender_features3(name):\n",
    "    features = {}\n",
    "    features[\"first_letter\"] = name[0].lower()\n",
    "    features[\"suffix1\"] = name[-1].lower()\n",
    "    features[\"suffix2\"] = name[-2:].lower()\n",
    "    for letter in 'abcdefghijklmnopqrstuvwxyz':\n",
    "        features[\"count({})\".format(letter)] = name.lower().count(letter)\n",
    "        features[\"has({})\".format(letter)] = (letter in name.lower())\n",
    "    return features\n",
    "\n",
    "gender_features3('John')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'count(a)': 0,\n",
       "  'count(b)': 0,\n",
       "  'count(c)': 0,\n",
       "  'count(d)': 0,\n",
       "  'count(e)': 0,\n",
       "  'count(f)': 0,\n",
       "  'count(g)': 0,\n",
       "  'count(h)': 0,\n",
       "  'count(i)': 2,\n",
       "  'count(j)': 0,\n",
       "  'count(k)': 0,\n",
       "  'count(l)': 3,\n",
       "  'count(m)': 0,\n",
       "  'count(n)': 0,\n",
       "  'count(o)': 0,\n",
       "  'count(p)': 0,\n",
       "  'count(q)': 0,\n",
       "  'count(r)': 0,\n",
       "  'count(s)': 0,\n",
       "  'count(t)': 0,\n",
       "  'count(u)': 0,\n",
       "  'count(v)': 0,\n",
       "  'count(w)': 0,\n",
       "  'count(x)': 0,\n",
       "  'count(y)': 0,\n",
       "  'count(z)': 0,\n",
       "  'first_letter': 'l',\n",
       "  'has(a)': False,\n",
       "  'has(b)': False,\n",
       "  'has(c)': False,\n",
       "  'has(d)': False,\n",
       "  'has(e)': False,\n",
       "  'has(f)': False,\n",
       "  'has(g)': False,\n",
       "  'has(h)': False,\n",
       "  'has(i)': True,\n",
       "  'has(j)': False,\n",
       "  'has(k)': False,\n",
       "  'has(l)': True,\n",
       "  'has(m)': False,\n",
       "  'has(n)': False,\n",
       "  'has(o)': False,\n",
       "  'has(p)': False,\n",
       "  'has(q)': False,\n",
       "  'has(r)': False,\n",
       "  'has(s)': False,\n",
       "  'has(t)': False,\n",
       "  'has(u)': False,\n",
       "  'has(v)': False,\n",
       "  'has(w)': False,\n",
       "  'has(x)': False,\n",
       "  'has(y)': False,\n",
       "  'has(z)': False,\n",
       "  'suffix1': 'i',\n",
       "  'suffix2': 'li'},\n",
       " 'female')"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "featuresets3 = [(gender_features3(n), gender) for (n, gender) in labeled_names]\n",
    "featuresets3[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.79"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set3, test_set3 = featuresets3[:TRAIN_SET_SIZE], featuresets3[TRAIN_SET_SIZE:]\n",
    "classifier3 = NaiveBayesClassifier.train(train_set3)\n",
    "round(accuracy(classifier3, test_set3), 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Most Informative Features\n",
      "                 suffix2 = 'na'           female : male   =     83.4 : 1.0\n",
      "                 suffix2 = 'ia'           female : male   =     79.6 : 1.0\n",
      "                 suffix1 = 'a'            female : male   =     34.1 : 1.0\n",
      "                 suffix2 = 'sa'           female : male   =     31.7 : 1.0\n",
      "                 suffix2 = 'rd'             male : female =     28.9 : 1.0\n",
      "                 suffix1 = 'k'              male : female =     28.0 : 1.0\n",
      "                 suffix2 = 'us'             male : female =     25.5 : 1.0\n",
      "                 suffix2 = 'ra'           female : male   =     23.0 : 1.0\n",
      "                 suffix2 = 'ld'             male : female =     21.1 : 1.0\n",
      "                 suffix2 = 'ta'           female : male   =     20.8 : 1.0\n",
      "                 suffix2 = 'do'             male : female =     20.6 : 1.0\n",
      "                 suffix2 = 'rt'             male : female =     19.3 : 1.0\n",
      "                 suffix1 = 'v'              male : female =     16.5 : 1.0\n",
      "                 suffix2 = 'io'             male : female =     16.4 : 1.0\n",
      "                 suffix2 = 'ka'           female : male   =     13.0 : 1.0\n"
     ]
    }
   ],
   "source": [
    "classifier3.show_most_informative_features(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Trying to use a maximum entropy classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The principle of **maximum entropy** states that the probability distribution which best represents the current state of knowledge is the one with largest entropy.\n",
    "\n",
    "The principle of maximum entropy is invoked when we have some piece(s) of information about a probability distribution, but not enough to characterize it completely—likely because we do not have the means or resources to do so. As an example, if all we know about a distribution is its average, we can imagine infinite shapes that yield a particular average. The principle of maximum entropy says that we should humbly choose the distribution that maximizes the amount of unpredictability contained in the distribution.\n",
    "\n",
    "Taking the idea to the extreme, it wouldn’t be scientific to choose a distribution that simply yields the average value 100% of the time.\n",
    "\n",
    "From all the models that fit our training data, the Maximum Entropy classifier selects the one which has the largest entropy. Due to the minimum assumptions that the Maximum Entropy classifier makes, it is usually used when we don’t know anything about the prior distributions and when it is unsafe to make any assumptions. Also, the maximum entropy classifier is used when we can’t assume the conditional independence of the features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  ==> Training (25 iterations)\n",
      "\n",
      "      Iteration    Log Likelihood    Accuracy\n",
      "      ---------------------------------------\n",
      "             1          -0.69315        0.369\n",
      "             2          -0.60190        0.631\n",
      "             3          -0.58085        0.631\n",
      "             4          -0.56149        0.635\n",
      "             5          -0.54373        0.663\n",
      "             6          -0.52748        0.698\n",
      "             7          -0.51261        0.729\n",
      "             8          -0.49902        0.752\n",
      "             9          -0.48659        0.765\n",
      "            10          -0.47520        0.774\n",
      "            11          -0.46475        0.784\n",
      "            12          -0.45514        0.789\n",
      "            13          -0.44630        0.794\n",
      "            14          -0.43815        0.797\n",
      "            15          -0.43062        0.800\n",
      "            16          -0.42364        0.804\n",
      "            17          -0.41716        0.805\n",
      "            18          -0.41115        0.806\n",
      "            19          -0.40554        0.806\n",
      "            20          -0.40031        0.808\n",
      "            21          -0.39542        0.811\n",
      "            22          -0.39085        0.811\n",
      "            23          -0.38656        0.813\n",
      "            24          -0.38252        0.814\n",
      "         Final          -0.37873        0.815\n"
     ]
    }
   ],
   "source": [
    "from nltk import MaxentClassifier\n",
    "\n",
    "me_classifier = MaxentClassifier.train(train_set3, max_iter=25) # max_iter has default value 100. In this example, the performance in terms of accuracy on the test set starts significantly improving beyond the previous model's at around 25 iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round(accuracy(me_classifier, test_set3), 2) # The accuracies above were on the training set so this is what matters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  -1.978 suffix2=='ia' and label is 'male'\n",
      "  -1.921 suffix2=='na' and label is 'male'\n",
      "  -1.515 suffix2=='sa' and label is 'male'\n",
      "  -1.463 suffix1=='a' and label is 'male'\n",
      "  -1.290 suffix2=='ra' and label is 'male'\n",
      "  -1.278 suffix1=='k' and label is 'female'\n",
      "  -1.197 suffix2=='rd' and label is 'female'\n",
      "  -1.169 suffix2=='do' and label is 'female'\n",
      "  -1.167 suffix2=='us' and label is 'female'\n",
      "  -1.166 suffix2=='ta' and label is 'male'\n"
     ]
    }
   ],
   "source": [
    "me_classifier.show_most_informative_features(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. More classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scikit-learn (sklearn) is a popular library which features various classification, regression and clustering algorithms including support vector machines, random forests, gradient boosting, k-means and DBSCAN.\n",
    "\n",
    "NLTK provides an API to quickly use sklearn classifiers in `nltk.classify.scikitlearn`. The other option is to import and use sklearn directly.\n",
    "\n",
    "For an example of integrating sklearn with NLTK, you can check out [this](https://www.kaggle.com/alvations/basic-nlp-with-nltk) notebook on Kaggle. Kaggle is a great website for NLP and machine learning in general, creating an account is highly recommended."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}