08-Classification-Exercise-SA-udemy-TF.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Classification Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll be working with some California Census Data, we'll be trying to use various features of an individual to predict what class of income they belogn in (>50k or <=50k). \n",
    "\n",
    "Here is some information about the data:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    "<thead>\n",
    "<tr>\n",
    "<th>Column Name</th>\n",
    "<th>Type</th>\n",
    "<th>Description</th>\n",
    "</tr>\n",
    "</thead>\n",
    "<tbody>\n",
    "<tr>\n",
    "<td>age</td>\n",
    "<td>Continuous</td>\n",
    "<td>The age of the individual</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>workclass</td>\n",
    "<td>Categorical</td>\n",
    "<td>The type of employer the  individual has (government,  military, private, etc.).</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>fnlwgt</td>\n",
    "<td>Continuous</td>\n",
    "<td>The number of people the census  takers believe that observation  represents (sample weight). This  variable will not be used.</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>education</td>\n",
    "<td>Categorical</td>\n",
    "<td>The highest level of education  achieved for that individual.</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>education_num</td>\n",
    "<td>Continuous</td>\n",
    "<td>The highest level of education in  numerical form.</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>marital_status</td>\n",
    "<td>Categorical</td>\n",
    "<td>Marital status of the individual.</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>occupation</td>\n",
    "<td>Categorical</td>\n",
    "<td>The occupation of the individual.</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>relationship</td>\n",
    "<td>Categorical</td>\n",
    "<td>Wife, Own-child, Husband,  Not-in-family, Other-relative,  Unmarried.</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>race</td>\n",
    "<td>Categorical</td>\n",
    "<td>White, Asian-Pac-Islander,  Amer-Indian-Eskimo, Other, Black.</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>gender</td>\n",
    "<td>Categorical</td>\n",
    "<td>Female, Male.</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>capital_gain</td>\n",
    "<td>Continuous</td>\n",
    "<td>Capital gains recorded.</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>capital_loss</td>\n",
    "<td>Continuous</td>\n",
    "<td>Capital Losses recorded.</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>hours_per_week</td>\n",
    "<td>Continuous</td>\n",
    "<td>Hours worked per week.</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>native_country</td>\n",
    "<td>Categorical</td>\n",
    "<td>Country of origin of the  individual.</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>income</td>\n",
    "<td>Categorical</td>\n",
    "<td>\"&gt;50K\" or \"&lt;=50K\", meaning  whether the person makes more  than \\$50,000 annually.</td>\n",
    "</tr>\n",
    "</tbody>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Follow the Directions in Bold. If you get stuck, check out the solutions lecture."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### THE DATA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Read in the census_data.csv data with pandas**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "data=pd.read_csv(\"census_data.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>education</th>\n",
       "      <th>education_num</th>\n",
       "      <th>marital_status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>gender</th>\n",
       "      <th>capital_gain</th>\n",
       "      <th>capital_loss</th>\n",
       "      <th>hours_per_week</th>\n",
       "      <th>native_country</th>\n",
       "      <th>income_bracket</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>32556</th>\n",
       "      <td>27</td>\n",
       "      <td>Private</td>\n",
       "      <td>Assoc-acdm</td>\n",
       "      <td>12</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Tech-support</td>\n",
       "      <td>Wife</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>38</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32557</th>\n",
       "      <td>40</td>\n",
       "      <td>Private</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32558</th>\n",
       "      <td>58</td>\n",
       "      <td>Private</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Widowed</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Unmarried</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32559</th>\n",
       "      <td>22</td>\n",
       "      <td>Private</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32560</th>\n",
       "      <td>52</td>\n",
       "      <td>Self-emp-inc</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Wife</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>15024</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       age      workclass    education  education_num       marital_status  \\\n",
       "32556   27        Private   Assoc-acdm             12   Married-civ-spouse   \n",
       "32557   40        Private      HS-grad              9   Married-civ-spouse   \n",
       "32558   58        Private      HS-grad              9              Widowed   \n",
       "32559   22        Private      HS-grad              9        Never-married   \n",
       "32560   52   Self-emp-inc      HS-grad              9   Married-civ-spouse   \n",
       "\n",
       "               occupation relationship    race   gender  capital_gain  \\\n",
       "32556        Tech-support         Wife   White   Female             0   \n",
       "32557   Machine-op-inspct      Husband   White     Male             0   \n",
       "32558        Adm-clerical    Unmarried   White   Female             0   \n",
       "32559        Adm-clerical    Own-child   White     Male             0   \n",
       "32560     Exec-managerial         Wife   White   Female         15024   \n",
       "\n",
       "       capital_loss  hours_per_week  native_country income_bracket  \n",
       "32556             0              38   United-States          <=50K  \n",
       "32557             0              40   United-States           >50K  \n",
       "32558             0              40   United-States          <=50K  \n",
       "32559             0              20   United-States          <=50K  \n",
       "32560             0              40   United-States           >50K  "
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** TensorFlow won't be able to understand strings as labels, you'll need to use pandas .apply() method to apply a custom function that converts them to 0s and 1s. This might be hard if you aren't very familiar with pandas, so feel free to take a peek at the solutions for this part.**\n",
    "\n",
    "** Convert the Label column to 0s and 1s instead of strings.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([' <=50K', ' >50K'], dtype=object)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['income_bracket'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "def label_fix(label):\n",
    "    if label==' <=50K':\n",
    "        return 0\n",
    "    else:\n",
    "        return 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['income_bracket']=data['income_bracket'].apply(label_fix)\n",
    "#census['income_bracket'] = census['income_bracket'].apply(label_fix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_data = data.drop('income_bracket',axis=1)\n",
    "y_labels = data['income_bracket']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Perform a Train Test Split on the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train, x_test, Y_train, Y_test = train_test_split(x_data, y_labels, test_size=0.3, random_state=101)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "#"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Feature Columns for tf.esitmator\n",
    "\n",
    "** Take note of categorical vs continuous values! **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['age', 'workclass', 'education', 'education_num', 'marital_status',\n",
       "       'occupation', 'relationship', 'race', 'gender', 'capital_gain',\n",
       "       'capital_loss', 'hours_per_week', 'native_country', 'income_bracket'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Import Tensorflow **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "#"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "** Create the tf.feature_columns for the categorical values. Use vocabulary lists or just use hash buckets. **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "gender = tf.feature_column.categorical_column_with_vocabulary_list(\"gender\", [\"Female\", \"Male\"])\n",
    "occupation = tf.feature_column.categorical_column_with_hash_bucket(\"occupation\", hash_bucket_size=1000)\n",
    "marital_status = tf.feature_column.categorical_column_with_hash_bucket(\"marital_status\", hash_bucket_size=1000)\n",
    "relationship = tf.feature_column.categorical_column_with_hash_bucket(\"relationship\", hash_bucket_size=1000)\n",
    "education = tf.feature_column.categorical_column_with_hash_bucket(\"education\", hash_bucket_size=1000)\n",
    "workclass = tf.feature_column.categorical_column_with_hash_bucket(\"workclass\", hash_bucket_size=1000)\n",
    "native_country = tf.feature_column.categorical_column_with_hash_bucket(\"native_country\", hash_bucket_size=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Create the continuous feature_columns for the continuous values using numeric_column **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "age = tf.feature_column.numeric_column(\"age\")\n",
    "education_num = tf.feature_column.numeric_column(\"education_num\")\n",
    "capital_gain = tf.feature_column.numeric_column(\"capital_gain\")\n",
    "capital_loss = tf.feature_column.numeric_column(\"capital_loss\")\n",
    "hours_per_week = tf.feature_column.numeric_column(\"hours_per_week\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Put all these variables into a single list with the variable name feat_cols **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "feat_list=[gender,occupation,marital_status,relationship,education,workclass,native_country,\n",
    "            age,education_num,capital_gain,capital_loss,hours_per_week]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Input Function\n",
    "\n",
    "** Batch_size is up to you. But do make sure to shuffle!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_fn=tf.estimator.inputs.pandas_input_fn(x=x_train,y=Y_train,batch_size=10,num_epochs=None,shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create your model with tf.estimator\n",
    "\n",
    "**Create a LinearClassifier.(If you want to use a DNNClassifier, keep in mind you'll need to create embedded columns out of the cateogrical feature that use strings, check out the previous lecture on this for more info.)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "WARNING:tensorflow:Using temporary folder as model directory: /var/folders/c3/88hk3vsx7vvbx2k7w_c2wwtshd1228/T/tmp4xwjkq9u\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/var/folders/c3/88hk3vsx7vvbx2k7w_c2wwtshd1228/T/tmp4xwjkq9u', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x122b974a8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
     ]
    }
   ],
   "source": [
    "model = tf.estimator.LinearClassifier(feature_columns=feat_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Train your model on the data, for at least 5000 steps. **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 1 into /var/folders/c3/88hk3vsx7vvbx2k7w_c2wwtshd1228/T/tmp4xwjkq9u/model.ckpt.\n",
      "INFO:tensorflow:loss = 6.931472, step = 1\n",
      "INFO:tensorflow:global_step/sec: 189.782\n",
      "INFO:tensorflow:loss = 4.429467, step = 101 (0.531 sec)\n",
      "INFO:tensorflow:global_step/sec: 402.143\n",
      "INFO:tensorflow:loss = 57.560986, step = 201 (0.246 sec)\n",
      "INFO:tensorflow:global_step/sec: 396.723\n",
      "INFO:tensorflow:loss = 4.5567513, step = 301 (0.252 sec)\n",
      "INFO:tensorflow:global_step/sec: 390.59\n",
      "INFO:tensorflow:loss = 1.8060349, step = 401 (0.257 sec)\n",
      "INFO:tensorflow:global_step/sec: 398.28\n",
      "INFO:tensorflow:loss = 3.2214663, step = 501 (0.249 sec)\n",
      "INFO:tensorflow:global_step/sec: 391.383\n",
      "INFO:tensorflow:loss = 4.193394, step = 601 (0.256 sec)\n",
      "INFO:tensorflow:global_step/sec: 380.871\n",
      "INFO:tensorflow:loss = 0.8870957, step = 701 (0.263 sec)\n",
      "INFO:tensorflow:global_step/sec: 398.99\n",
      "INFO:tensorflow:loss = 4.6227093, step = 801 (0.251 sec)\n",
      "INFO:tensorflow:global_step/sec: 400.312\n",
      "INFO:tensorflow:loss = 4.9870815, step = 901 (0.250 sec)\n",
      "INFO:tensorflow:global_step/sec: 388.314\n",
      "INFO:tensorflow:loss = 1.2140492, step = 1001 (0.257 sec)\n",
      "INFO:tensorflow:global_step/sec: 219.279\n",
      "INFO:tensorflow:loss = 8.2452965, step = 1101 (0.458 sec)\n",
      "INFO:tensorflow:global_step/sec: 307.924\n",
      "INFO:tensorflow:loss = 3.3655176, step = 1201 (0.323 sec)\n",
      "INFO:tensorflow:global_step/sec: 366.24\n",
      "INFO:tensorflow:loss = 3.9630656, step = 1301 (0.273 sec)\n",
      "INFO:tensorflow:global_step/sec: 409.339\n",
      "INFO:tensorflow:loss = 2.46638, step = 1401 (0.245 sec)\n",
      "INFO:tensorflow:global_step/sec: 353.353\n",
      "INFO:tensorflow:loss = 2.0904217, step = 1501 (0.283 sec)\n",
      "INFO:tensorflow:global_step/sec: 398.994\n",
      "INFO:tensorflow:loss = 23.605413, step = 1601 (0.251 sec)\n",
      "INFO:tensorflow:global_step/sec: 411.735\n",
      "INFO:tensorflow:loss = 2.9920485, step = 1701 (0.243 sec)\n",
      "INFO:tensorflow:global_step/sec: 406.755\n",
      "INFO:tensorflow:loss = 14.577152, step = 1801 (0.246 sec)\n",
      "INFO:tensorflow:global_step/sec: 300.151\n",
      "INFO:tensorflow:loss = 14.614017, step = 1901 (0.338 sec)\n",
      "INFO:tensorflow:global_step/sec: 279.727\n",
      "INFO:tensorflow:loss = 6.862772, step = 2001 (0.355 sec)\n",
      "INFO:tensorflow:global_step/sec: 338.059\n",
      "INFO:tensorflow:loss = 5.362343, step = 2101 (0.297 sec)\n",
      "INFO:tensorflow:global_step/sec: 410.352\n",
      "INFO:tensorflow:loss = 20.00618, step = 2201 (0.243 sec)\n",
      "INFO:tensorflow:global_step/sec: 373.141\n",
      "INFO:tensorflow:loss = 0.7059752, step = 2301 (0.268 sec)\n",
      "INFO:tensorflow:global_step/sec: 373.826\n",
      "INFO:tensorflow:loss = 1.9181341, step = 2401 (0.268 sec)\n",
      "INFO:tensorflow:global_step/sec: 416.891\n",
      "INFO:tensorflow:loss = 13.778948, step = 2501 (0.238 sec)\n",
      "INFO:tensorflow:global_step/sec: 409.232\n",
      "INFO:tensorflow:loss = 10.927493, step = 2601 (0.244 sec)\n",
      "INFO:tensorflow:global_step/sec: 421.669\n",
      "INFO:tensorflow:loss = 12.499138, step = 2701 (0.237 sec)\n",
      "INFO:tensorflow:global_step/sec: 404.778\n",
      "INFO:tensorflow:loss = 39.62496, step = 2801 (0.245 sec)\n",
      "INFO:tensorflow:global_step/sec: 392.748\n",
      "INFO:tensorflow:loss = 9.031824, step = 2901 (0.255 sec)\n",
      "INFO:tensorflow:global_step/sec: 394.272\n",
      "INFO:tensorflow:loss = 1.0561242, step = 3001 (0.254 sec)\n",
      "INFO:tensorflow:global_step/sec: 397.506\n",
      "INFO:tensorflow:loss = 1.0734874, step = 3101 (0.252 sec)\n",
      "INFO:tensorflow:global_step/sec: 390.229\n",
      "INFO:tensorflow:loss = 7.784442, step = 3201 (0.256 sec)\n",
      "INFO:tensorflow:global_step/sec: 405.101\n",
      "INFO:tensorflow:loss = 3.6667929, step = 3301 (0.247 sec)\n",
      "INFO:tensorflow:global_step/sec: 390.149\n",
      "INFO:tensorflow:loss = 1.9981585, step = 3401 (0.257 sec)\n",
      "INFO:tensorflow:global_step/sec: 403.268\n",
      "INFO:tensorflow:loss = 0.7246193, step = 3501 (0.248 sec)\n",
      "INFO:tensorflow:global_step/sec: 409.425\n",
      "INFO:tensorflow:loss = 4.0812, step = 3601 (0.244 sec)\n",
      "INFO:tensorflow:global_step/sec: 396.697\n",
      "INFO:tensorflow:loss = 0.7211969, step = 3701 (0.252 sec)\n",
      "INFO:tensorflow:global_step/sec: 410.253\n",
      "INFO:tensorflow:loss = 65.47055, step = 3801 (0.244 sec)\n",
      "INFO:tensorflow:global_step/sec: 391.31\n",
      "INFO:tensorflow:loss = 2.9494262, step = 3901 (0.256 sec)\n",
      "INFO:tensorflow:global_step/sec: 394.847\n",
      "INFO:tensorflow:loss = 0.9497367, step = 4001 (0.254 sec)\n",
      "INFO:tensorflow:global_step/sec: 389.206\n",
      "INFO:tensorflow:loss = 1.5431234, step = 4101 (0.256 sec)\n",
      "INFO:tensorflow:global_step/sec: 382.133\n",
      "INFO:tensorflow:loss = 39.03382, step = 4201 (0.262 sec)\n",
      "INFO:tensorflow:global_step/sec: 374.813\n",
      "INFO:tensorflow:loss = 16.60341, step = 4301 (0.268 sec)\n",
      "INFO:tensorflow:global_step/sec: 391.203\n",
      "INFO:tensorflow:loss = 1.8292488, step = 4401 (0.254 sec)\n",
      "INFO:tensorflow:global_step/sec: 396.968\n",
      "INFO:tensorflow:loss = 4.80962, step = 4501 (0.252 sec)\n",
      "INFO:tensorflow:global_step/sec: 347.176\n",
      "INFO:tensorflow:loss = 4.463552, step = 4601 (0.288 sec)\n",
      "INFO:tensorflow:global_step/sec: 389.918\n",
      "INFO:tensorflow:loss = 2.1998167, step = 4701 (0.256 sec)\n",
      "INFO:tensorflow:global_step/sec: 369.907\n",
      "INFO:tensorflow:loss = 10.89747, step = 4801 (0.270 sec)\n",
      "INFO:tensorflow:global_step/sec: 384.131\n",
      "INFO:tensorflow:loss = 86.826256, step = 4901 (0.260 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 5000 into /var/folders/c3/88hk3vsx7vvbx2k7w_c2wwtshd1228/T/tmp4xwjkq9u/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 9.648349.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.estimator.canned.linear.LinearClassifier at 0x117f2f6a0>"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.train(input_fn, steps=5000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation\n",
    "\n",
    "** Create a prediction input function. Remember to only supprt X_test data and keep shuffle=False. **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_fn=tf.estimator.inputs.pandas_input_fn(x=x_test, batch_size=len(x_test), num_epochs=1,shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Use model.predict() and pass in your input function. This will produce a generator of predictions, which you can then transform into a list, with list() **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from /var/folders/c3/88hk3vsx7vvbx2k7w_c2wwtshd1228/T/tmp4xwjkq9u/model.ckpt-5000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n"
     ]
    }
   ],
   "source": [
    "predictions = model.predict(input_fn=eval_fn)\n",
    "lpred=list(predictions)\n",
    "#results = model.evaluate(eval_input_func)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Each item in your list will look like this: **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'logits': array([-1.4367727], dtype=float32),\n",
       " 'logistic': array([0.19204561], dtype=float32),\n",
       " 'probabilities': array([0.8079544 , 0.19204561], dtype=float32),\n",
       " 'class_ids': array([0]),\n",
       " 'classes': array([b'0'], dtype=object)}"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lpred[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Create a list of only the class_ids key values from the prediction list of dictionaries, these are the predictions you will use to compare against the real y_test values. **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_preds = []\n",
    "for pred in lpred:\n",
    "    final_preds.append(pred['class_ids'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 0, 0, 0, 1, 0, 0, 0, 0, 0]"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_preds[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Import classification_report from sklearn.metrics and then see if you can figure out how to use it to easily get a full report of your model's performance on the test data. **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.88      0.91      0.89      7436\n",
      "          1       0.68      0.59      0.63      2333\n",
      "\n",
      "avg / total       0.83      0.84      0.83      9769\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(Y_test,final_preds))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "#DNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "WARNING:tensorflow:Using temporary folder as model directory: /var/folders/c3/88hk3vsx7vvbx2k7w_c2wwtshd1228/T/tmpwje_gpab\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/var/folders/c3/88hk3vsx7vvbx2k7w_c2wwtshd1228/T/tmpwje_gpab', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x123065b70>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
     ]
    }
   ],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedded_group_column_gender = tf.feature_column.embedding_column(gender, dimension=2)\n",
    "embedded_group_column_occuption = tf.feature_column.embedding_column(occupation, dimension=15)\n",
    "embedded_group_column_ms = tf.feature_column.embedding_column(marital_status, dimension=7)\n",
    "embedded_group_column_relation = tf.feature_column.embedding_column(relationship, dimension=6)\n",
    "embedded_group_column_edu = tf.feature_column.embedding_column(education, dimension=16)\n",
    "embedded_group_column_workc = tf.feature_column.embedding_column(workclass, dimension=9)\n",
    "embedded_group_column_nativec = tf.feature_column.embedding_column(native_country, dimension=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([' United-States', ' Cuba', ' Jamaica', ' India', ' ?', ' Mexico',\n",
       "       ' South', ' Puerto-Rico', ' Honduras', ' England', ' Canada',\n",
       "       ' Germany', ' Iran', ' Philippines', ' Italy', ' Poland',\n",
       "       ' Columbia', ' Cambodia', ' Thailand', ' Ecuador', ' Laos',\n",
       "       ' Taiwan', ' Haiti', ' Portugal', ' Dominican-Republic',\n",
       "       ' El-Salvador', ' France', ' Guatemala', ' China', ' Japan',\n",
       "       ' Yugoslavia', ' Peru', ' Outlying-US(Guam-USVI-etc)', ' Scotland',\n",
       "       ' Trinadad&Tobago', ' Greece', ' Nicaragua', ' Vietnam', ' Hong',\n",
       "       ' Ireland', ' Hungary', ' Holand-Netherlands'], dtype=object)"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['native_country'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "feat_list=[embedded_group_column_gender,embedded_group_column_occuption,embedded_group_column_ms,embedded_group_column_relation,\n",
    "           embedded_group_column_edu,embedded_group_column_workc,embedded_group_column_nativec,\n",
    "            age,education_num,capital_gain,capital_loss,hours_per_week]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_func = tf.estimator.inputs.pandas_input_fn(x=x_train,y=Y_train,batch_size=10,num_epochs=None,shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "WARNING:tensorflow:Using temporary folder as model directory: /var/folders/c3/88hk3vsx7vvbx2k7w_c2wwtshd1228/T/tmpvan_sspu\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/var/folders/c3/88hk3vsx7vvbx2k7w_c2wwtshd1228/T/tmpvan_sspu', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x122bac320>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
     ]
    }
   ],
   "source": [
    "dnn_model = tf.estimator.DNNClassifier(hidden_units=[10,10,10],feature_columns=feat_list,n_classes=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 1 into /var/folders/c3/88hk3vsx7vvbx2k7w_c2wwtshd1228/T/tmpvan_sspu/model.ckpt.\n",
      "INFO:tensorflow:loss = 391.59756, step = 1\n",
      "INFO:tensorflow:Saving checkpoints for 100 into /var/folders/c3/88hk3vsx7vvbx2k7w_c2wwtshd1228/T/tmpvan_sspu/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 6.100886.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.estimator.canned.dnn.DNNClassifier at 0x122bacb38>"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dnn_model.train(input_fn=input_func,steps=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_input_func = tf.estimator.inputs.pandas_input_fn(\n",
    "      x=x_test,\n",
    "      y=Y_test,\n",
    "      batch_size=10,\n",
    "      num_epochs=1,\n",
    "      shuffle=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2018-07-07-23:15:29\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from /var/folders/c3/88hk3vsx7vvbx2k7w_c2wwtshd1228/T/tmpvan_sspu/model.ckpt-100\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2018-07-07-23:15:31\n",
      "INFO:tensorflow:Saving dict for global step 100: accuracy = 0.7854437, accuracy_baseline = 0.7611833, auc = 0.6394565, auc_precision_recall = 0.46747348, average_loss = 0.9464054, global_step = 100, label/mean = 0.23881666, loss = 9.463085, precision = 0.6453988, prediction/mean = 0.2821161, recall = 0.22546078\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'accuracy': 0.7854437,\n",
       " 'accuracy_baseline': 0.7611833,\n",
       " 'auc': 0.6394565,\n",
       " 'auc_precision_recall': 0.46747348,\n",
       " 'average_loss': 0.9464054,\n",
       " 'label/mean': 0.23881666,\n",
       " 'loss': 9.463085,\n",
       " 'precision': 0.6453988,\n",
       " 'prediction/mean': 0.2821161,\n",
       " 'recall': 0.22546078,\n",
       " 'global_step': 100}"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dnn_model.evaluate(eval_input_func)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_fn = tf.estimator.inputs.pandas_input_fn(x=x_test,batch_size=len(x_test),shuffle=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from /var/folders/c3/88hk3vsx7vvbx2k7w_c2wwtshd1228/T/tmpvan_sspu/model.ckpt-100\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n"
     ]
    }
   ],
   "source": [
    "predictions=list(dnn_model.predict(pred_fn))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'logits': array([-1.4742498], dtype=float32),\n",
       " 'logistic': array([0.18629752], dtype=float32),\n",
       " 'probabilities': array([0.8137025 , 0.18629752], dtype=float32),\n",
       " 'class_ids': array([0]),\n",
       " 'classes': array([b'0'], dtype=object)}"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_preds = []\n",
    "for pred in predictions:\n",
    "    final_preds.append(pred['class_ids'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 0, 0, 0, 1, 0, 0, 0, 0, 0]"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_preds[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.80      0.96      0.87      7436\n",
      "          1       0.65      0.23      0.33      2333\n",
      "\n",
      "avg / total       0.76      0.79      0.74      9769\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(Y_test,final_preds))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'\n",
    "gender = tf.feature_column.categorical_column_with_vocabulary_list(\"gender\", [\"Female\", \"Male\"])\n",
    "occupation = tf.feature_column.categorical_column_with_hash_bucket(\"occupation\", hash_bucket_size=1000)\n",
    "marital_status = tf.feature_column.categorical_column_with_hash_bucket(\"marital_status\", hash_bucket_size=1000)\n",
    "relationship = tf.feature_column.categorical_column_with_hash_bucket(\"relationship\", hash_bucket_size=1000)\n",
    "education = tf.feature_column.categorical_column_with_hash_bucket(\"education\", hash_bucket_size=1000)\n",
    "workclass = tf.feature_column.categorical_column_with_hash_bucket(\"workclass\", hash_bucket_size=1000)\n",
    "native_country = tf.feature_column.categorical_column_with_hash_bucket(\"native_country\", hash_bucket_size=1000)\n",
    "'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Great Job!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}