DW07_Missing_Value.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bfca977b",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Missing-values\" data-toc-modified-id=\"Missing-values-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Missing values</a></span></li><li><span><a href=\"#Check-missing-values\" data-toc-modified-id=\"Check-missing-values-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Check missing values</a></span></li><li><span><a href=\"#Investigate-with-missing-values-for-ages\" data-toc-modified-id=\"Investigate-with-missing-values-for-ages-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Investigate with missing values for ages</a></span><ul class=\"toc-item\"><li><span><a href=\"#Discussion:\" data-toc-modified-id=\"Discussion:-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>Discussion:</a></span></li><li><span><a href=\"#Discussion:\" data-toc-modified-id=\"Discussion:-3.2\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>Discussion:</a></span><ul class=\"toc-item\"><li><span><a href=\"#What-does-age-=-0.17-mean?\" data-toc-modified-id=\"What-does-age-=-0.17-mean?-3.2.1\"><span class=\"toc-item-num\">3.2.1&nbsp;&nbsp;</span>What does age = 0.17 mean?</a></span></li></ul></li><li><span><a href=\"#Let's-look-at-other-factors,-e.g.-names-can-give-a-clue-because-most-passengers-seem-to-have-a-title-(e.g.-'Mrs')\" data-toc-modified-id=\"Let's-look-at-other-factors,-e.g.-names-can-give-a-clue-because-most-passengers-seem-to-have-a-title-(e.g.-'Mrs')-3.3\"><span class=\"toc-item-num\">3.3&nbsp;&nbsp;</span>Let's look at other factors, e.g. names can give a clue because most passengers seem to have a title (e.g. 'Mrs')</a></span></li><li><span><a href=\"#Discussion:\" data-toc-modified-id=\"Discussion:-3.4\"><span class=\"toc-item-num\">3.4&nbsp;&nbsp;</span>Discussion:</a></span></li><li><span><a href=\"#That-worked-really-well,-we-have-titles-for-everyone-and-there-are-only-a-few-strange-ones:\" data-toc-modified-id=\"That-worked-really-well,-we-have-titles-for-everyone-and-there-are-only-a-few-strange-ones:-3.5\"><span class=\"toc-item-num\">3.5&nbsp;&nbsp;</span>That worked really well, we have titles for everyone and there are only a few strange ones:</a></span><ul class=\"toc-item\"><li><span><a href=\"#So-what-can-we-say-about-'Master'?-Usually-refers-to-boys?\" data-toc-modified-id=\"So-what-can-we-say-about-'Master'?-Usually-refers-to-boys?-3.5.1\"><span class=\"toc-item-num\">3.5.1&nbsp;&nbsp;</span>So what can we say about 'Master'? Usually refers to boys?</a></span></li><li><span><a href=\"#Dr,-Rev,-Col,-Major,-Capt,-Sir,-Don,-would-all-be-adults?\" data-toc-modified-id=\"Dr,-Rev,-Col,-Major,-Capt,-Sir,-Don,-would-all-be-adults?-3.5.2\"><span class=\"toc-item-num\">3.5.2&nbsp;&nbsp;</span>Dr, Rev, Col, Major, Capt, Sir, Don, would all be adults?</a></span></li><li><span><a href=\"#First-let's-check-distribution-of-missing-values-across-different-titles\" data-toc-modified-id=\"First-let's-check-distribution-of-missing-values-across-different-titles-3.5.3\"><span class=\"toc-item-num\">3.5.3&nbsp;&nbsp;</span>First let's check distribution of missing values across different titles</a></span></li></ul></li><li><span><a href=\"#We-will-first-investigate-missing-values-in-doctors\" data-toc-modified-id=\"We-will-first-investigate-missing-values-in-doctors-3.6\"><span class=\"toc-item-num\">3.6&nbsp;&nbsp;</span>We will first investigate missing values in doctors</a></span><ul class=\"toc-item\"><li><span><a href=\"#What's-the-average-age-of-Drs?\" data-toc-modified-id=\"What's-the-average-age-of-Drs?-3.6.1\"><span class=\"toc-item-num\">3.6.1&nbsp;&nbsp;</span>What's the average age of Drs?</a></span></li></ul></li></ul></li><li><span><a href=\"#Missing-values:-ages-for-boys\" data-toc-modified-id=\"Missing-values:-ages-for-boys-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Missing values: ages for boys</a></span></li><li><span><a href=\"#Mean-imputation-of-age-by-title\" data-toc-modified-id=\"Mean-imputation-of-age-by-title-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Mean imputation of age by title</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d8fb00b",
   "metadata": {},
   "source": [
    "# Missing values and imputation\n",
    "\n",
    "\n",
    "## Missing values\n",
    "\n",
    "**Source of missing values:**\n",
    "\n",
    "* Data Extraction: It is possible that there are problems with extraction process. In such cases, we should double-check for correct data with data guardians. Some hashing procedures can also be used to make sure that data extraction is correct. Errors at data extraction stage are typically easy to find and can be corrected easily as well.\n",
    "* Data collection: These errors occur at time of data collection and are harder to correct. They can be categorized in four types:\n",
    ">* Missing completely at random: This is a case when the probability of missing variable is same for all observations. For example: respondents of data collection process decide that they will declare their earning after tossing a fair coin. If an head occurs, respondent declares his / her earnings & vice versa. Here each observation has equal chance of missing value.\n",
    ">* Missing at random: This is a case when variable is missing at random and missing ratio varies for different values / level of other input variables. For example: We are collecting data for age and female has higher missing value compare to male.\n",
    ">* Missing that depends on unobserved predictors: This is a case when the missing values are not random and are related to the unobserved input variable. For example: In a medical study, if a particular diagnostic causes discomfort, then there is higher chance of drop out from the study. This missing value is not at random unless we have included “discomfort” as an input variable for all patients.\n",
    ">* Missing that depends on the missing value itself: This is a case when the probability of missing value is directly correlated with missing value itself. For example: People with higher or lower income are likely to provide non-response to their earning.\n",
    "\n",
    "We are going to use the Titanic dataset again to explore missing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "29f6f7de",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T06:56:15.948161Z",
     "start_time": "2022-05-03T06:56:15.522805Z"
    }
   },
   "outputs": [],
   "source": [
    "#import library \n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9498b185",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T06:59:01.460753Z",
     "start_time": "2022-05-03T06:59:01.442455Z"
    }
   },
   "outputs": [],
   "source": [
    "#load the data \n",
    "df1 = pd.read_csv('data/titanic passenger list.csv') "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b542fd2a",
   "metadata": {},
   "source": [
    "We would normally check some data basics before analysis, such as \n",
    "* length (rows)\n",
    "* width (or columns)\n",
    "* column types\n",
    "* basic stats\n",
    "* head & tail of data\n",
    "\n",
    "We will skip this step in this note."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecc210d3",
   "metadata": {},
   "source": [
    "## Check missing values \n",
    "\n",
    "\n",
    "** info()**\n",
    "* tells us total number of records\n",
    "* gives us details on datatype, 'object' here means string, the others are int and float\n",
    "* also include see 'non-null' observations for each data column\n",
    "\n",
    "** describe() **by default tabulates numeric data, which report on:\n",
    "* descriptive statistics \n",
    "* number of non-null observations in \"count\"\n",
    "* with \"include='all'\" will include all columns and \"include=['O']\" will include objects\n",
    "\n",
    "**.isnull().sum()***\n",
    "* will return sum of missing values for each column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2b2f7eec",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:40:32.465581Z",
     "start_time": "2022-05-03T08:40:32.449965Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1309 entries, 0 to 1308\n",
      "Data columns (total 14 columns):\n",
      " #   Column     Non-Null Count  Dtype  \n",
      "---  ------     --------------  -----  \n",
      " 0   pclass     1309 non-null   int64  \n",
      " 1   survived   1309 non-null   int64  \n",
      " 2   name       1309 non-null   object \n",
      " 3   sex        1309 non-null   object \n",
      " 4   age        1046 non-null   float64\n",
      " 5   sibsp      1309 non-null   int64  \n",
      " 6   parch      1309 non-null   int64  \n",
      " 7   ticket     1309 non-null   object \n",
      " 8   fare       1308 non-null   float64\n",
      " 9   cabin      295 non-null    object \n",
      " 10  embarked   1307 non-null   object \n",
      " 11  boat       486 non-null    object \n",
      " 12  body       121 non-null    float64\n",
      " 13  home.dest  745 non-null    object \n",
      "dtypes: float64(3), int64(4), object(7)\n",
      "memory usage: 143.3+ KB\n"
     ]
    }
   ],
   "source": [
    "df1.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fefb82f4",
   "metadata": {},
   "source": [
    "There are 1309 x 14 records and there are many missing values in \"age\",\"fare\" (one record) \"cabin\", \"boat\" \"body\" and \"home.dest\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "98501567",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:41:08.815943Z",
     "start_time": "2022-05-03T08:41:08.774000Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pclass</th>\n",
       "      <th>survived</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>body</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1309.000000</td>\n",
       "      <td>1309.000000</td>\n",
       "      <td>1046.000000</td>\n",
       "      <td>1309.000000</td>\n",
       "      <td>1309.000000</td>\n",
       "      <td>1308.000000</td>\n",
       "      <td>121.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>2.294882</td>\n",
       "      <td>0.381971</td>\n",
       "      <td>29.881138</td>\n",
       "      <td>0.498854</td>\n",
       "      <td>0.385027</td>\n",
       "      <td>33.295479</td>\n",
       "      <td>160.809917</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.837836</td>\n",
       "      <td>0.486055</td>\n",
       "      <td>14.413493</td>\n",
       "      <td>1.041658</td>\n",
       "      <td>0.865560</td>\n",
       "      <td>51.758668</td>\n",
       "      <td>97.696922</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.170000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>21.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7.895800</td>\n",
       "      <td>72.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>28.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>14.454200</td>\n",
       "      <td>155.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>39.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>31.275000</td>\n",
       "      <td>256.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>512.329200</td>\n",
       "      <td>328.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            pclass     survived          age        sibsp        parch  \\\n",
       "count  1309.000000  1309.000000  1046.000000  1309.000000  1309.000000   \n",
       "mean      2.294882     0.381971    29.881138     0.498854     0.385027   \n",
       "std       0.837836     0.486055    14.413493     1.041658     0.865560   \n",
       "min       1.000000     0.000000     0.170000     0.000000     0.000000   \n",
       "25%       2.000000     0.000000    21.000000     0.000000     0.000000   \n",
       "50%       3.000000     0.000000    28.000000     0.000000     0.000000   \n",
       "75%       3.000000     1.000000    39.000000     1.000000     0.000000   \n",
       "max       3.000000     1.000000    80.000000     8.000000     9.000000   \n",
       "\n",
       "              fare        body  \n",
       "count  1308.000000  121.000000  \n",
       "mean     33.295479  160.809917  \n",
       "std      51.758668   97.696922  \n",
       "min       0.000000    1.000000  \n",
       "25%       7.895800   72.000000  \n",
       "50%      14.454200  155.000000  \n",
       "75%      31.275000  256.000000  \n",
       "max     512.329200  328.000000  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f5cdf82e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:41:20.031331Z",
     "start_time": "2022-05-03T08:41:20.002139Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>sex</th>\n",
       "      <th>ticket</th>\n",
       "      <th>cabin</th>\n",
       "      <th>embarked</th>\n",
       "      <th>boat</th>\n",
       "      <th>home.dest</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1309</td>\n",
       "      <td>1309</td>\n",
       "      <td>1309</td>\n",
       "      <td>295</td>\n",
       "      <td>1307</td>\n",
       "      <td>486</td>\n",
       "      <td>745</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>1307</td>\n",
       "      <td>2</td>\n",
       "      <td>929</td>\n",
       "      <td>186</td>\n",
       "      <td>3</td>\n",
       "      <td>27</td>\n",
       "      <td>369</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>Connolly, Miss. Kate</td>\n",
       "      <td>male</td>\n",
       "      <td>CA. 2343</td>\n",
       "      <td>C23 C25 C27</td>\n",
       "      <td>S</td>\n",
       "      <td>13</td>\n",
       "      <td>New York, NY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>2</td>\n",
       "      <td>843</td>\n",
       "      <td>11</td>\n",
       "      <td>6</td>\n",
       "      <td>914</td>\n",
       "      <td>39</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        name   sex    ticket        cabin embarked boat  \\\n",
       "count                   1309  1309      1309          295     1307  486   \n",
       "unique                  1307     2       929          186        3   27   \n",
       "top     Connolly, Miss. Kate  male  CA. 2343  C23 C25 C27        S   13   \n",
       "freq                       2   843        11            6      914   39   \n",
       "\n",
       "           home.dest  \n",
       "count            745  \n",
       "unique           369  \n",
       "top     New York, NY  \n",
       "freq              64  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1.describe(include=['O']) # 'O' for Objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4a4bc3eb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:41:37.073796Z",
     "start_time": "2022-05-03T08:41:37.064194Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pclass          0\n",
       "survived        0\n",
       "name            0\n",
       "sex             0\n",
       "age           263\n",
       "sibsp           0\n",
       "parch           0\n",
       "ticket          0\n",
       "fare            1\n",
       "cabin        1014\n",
       "embarked        2\n",
       "boat          823\n",
       "body         1188\n",
       "home.dest     564\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d32d6422",
   "metadata": {},
   "source": [
    "## Investigate with missing values for ages\n",
    "\n",
    "Titanic data is commonly used for predictive analytics, e.g. remove the 'survived' column and feed the other data into an algorithm to see if there are any predictors for survival, pclass was a factor, also gender, age another (and if you ever go on a cruise ship change your name to \"Womenandchildren\").\n",
    "\n",
    "Age are mostly missing. Should we just delete or do something basic like:  \n",
    "\n",
    "meanAge = np.mean(df1.age)\n",
    "\n",
    "df1.age = df1.age.fillna(meanAge) \n",
    "\n",
    "### Discussion:\n",
    "\n",
    "* **What's wrong with this approach?**\n",
    "\n",
    "* **What is the average age anyway?**\n",
    "\n",
    "*  **What is the average age for males, for females? Is that an improvement?**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "cbac9bb7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:42:47.856973Z",
     "start_time": "2022-05-03T08:42:47.849777Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "29.881137667304014"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(df1.age)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "aef57927",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:42:54.207916Z",
     "start_time": "2022-05-03T08:42:54.196354Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "24.00    47\n",
       "22.00    43\n",
       "21.00    41\n",
       "30.00    40\n",
       "18.00    39\n",
       "         ..\n",
       "0.33      1\n",
       "22.50     1\n",
       "70.50     1\n",
       "0.67      1\n",
       "26.50     1\n",
       "Name: age, Length: 98, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# age - we know there are some missing, let's dig deeper\n",
    "df1['age'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "04678b9c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:43:11.409373Z",
     "start_time": "2022-05-03T08:43:11.400735Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([29.  ,  0.92,  2.  , 30.  , 25.  , 48.  , 63.  , 39.  , 53.  ,\n",
       "       71.  , 47.  , 18.  , 24.  , 26.  , 80.  ,   nan, 50.  , 32.  ,\n",
       "       36.  , 37.  , 42.  , 19.  , 35.  , 28.  , 45.  , 40.  , 58.  ,\n",
       "       22.  , 41.  , 44.  , 59.  , 60.  , 33.  , 17.  , 11.  , 14.  ,\n",
       "       49.  , 76.  , 46.  , 27.  , 64.  , 55.  , 70.  , 38.  , 51.  ,\n",
       "       31.  ,  4.  , 54.  , 23.  , 43.  , 52.  , 16.  , 32.5 , 21.  ,\n",
       "       15.  , 65.  , 28.5 , 45.5 , 56.  , 13.  , 61.  , 34.  ,  6.  ,\n",
       "       57.  , 62.  , 67.  ,  1.  , 12.  , 20.  ,  0.83,  8.  ,  0.67,\n",
       "        7.  ,  3.  , 36.5 , 18.5 ,  5.  , 66.  ,  9.  ,  0.75, 70.5 ,\n",
       "       22.5 ,  0.33,  0.17, 40.5 , 10.  , 23.5 , 34.5 , 20.5 , 30.5 ,\n",
       "       55.5 , 38.5 , 14.5 , 24.5 , 60.5 , 74.  ,  0.42, 11.5 , 26.5 ])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# mostly younger, 20s to 30s\n",
    "df1['age'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c4f3a24",
   "metadata": {},
   "source": [
    "Notice the 'nan' above, it doesn't occur in counts() \n",
    "\n",
    "### Discussion:\n",
    "*  How many 'NaN' values for age are there?\n",
    "*  How many 0?\n",
    "*  How many < 1?\n",
    "*  What does 'NaN' mean?\n",
    "*  Is NaN == NaN true in Python?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e9f25ec5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:43:51.349529Z",
     "start_time": "2022-05-03T08:43:51.328845Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pclass</th>\n",
       "      <th>survived</th>\n",
       "      <th>name</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>ticket</th>\n",
       "      <th>fare</th>\n",
       "      <th>cabin</th>\n",
       "      <th>embarked</th>\n",
       "      <th>boat</th>\n",
       "      <th>body</th>\n",
       "      <th>home.dest</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Baumann, Mr. John D</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17318</td>\n",
       "      <td>25.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>New York, NY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Bradley, Mr. George (\"George Arthur Brayton\")</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>111427</td>\n",
       "      <td>26.5500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Los Angeles, CA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Brewe, Dr. Arthur Jackson</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>112379</td>\n",
       "      <td>39.6000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Philadelphia, PA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Cairns, Mr. Alexander</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>113798</td>\n",
       "      <td>31.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cassebeer, Mrs. Henry Arthur Jr (Eleanor Genev...</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>17770</td>\n",
       "      <td>27.7208</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>New York, NY</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    pclass  survived                                               name  \\\n",
       "15       1         0                                Baumann, Mr. John D   \n",
       "37       1         1      Bradley, Mr. George (\"George Arthur Brayton\")   \n",
       "40       1         0                          Brewe, Dr. Arthur Jackson   \n",
       "46       1         0                              Cairns, Mr. Alexander   \n",
       "59       1         1  Cassebeer, Mrs. Henry Arthur Jr (Eleanor Genev...   \n",
       "\n",
       "       sex  age  sibsp  parch    ticket     fare cabin embarked boat  body  \\\n",
       "15    male  NaN      0      0  PC 17318  25.9250   NaN        S  NaN   NaN   \n",
       "37    male  NaN      0      0    111427  26.5500   NaN        S    9   NaN   \n",
       "40    male  NaN      0      0    112379  39.6000   NaN        C  NaN   NaN   \n",
       "46    male  NaN      0      0    113798  31.0000   NaN        S  NaN   NaN   \n",
       "59  female  NaN      0      0     17770  27.7208   NaN        C    5   NaN   \n",
       "\n",
       "           home.dest  \n",
       "15      New York, NY  \n",
       "37   Los Angeles, CA  \n",
       "40  Philadelphia, PA  \n",
       "46               NaN  \n",
       "59      New York, NY  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1[df1['age'].isnull()].head() # look at some"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "23759bc0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:43:58.959863Z",
     "start_time": "2022-05-03T08:43:58.939522Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pclass</th>\n",
       "      <th>survived</th>\n",
       "      <th>name</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>ticket</th>\n",
       "      <th>fare</th>\n",
       "      <th>cabin</th>\n",
       "      <th>embarked</th>\n",
       "      <th>boat</th>\n",
       "      <th>body</th>\n",
       "      <th>home.dest</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Allison, Master. Hudson Trevor</td>\n",
       "      <td>male</td>\n",
       "      <td>0.92</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>113781</td>\n",
       "      <td>151.5500</td>\n",
       "      <td>C22 C26</td>\n",
       "      <td>S</td>\n",
       "      <td>11</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Montreal, PQ / Chesterville, ON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>359</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>Caldwell, Master. Alden Gates</td>\n",
       "      <td>male</td>\n",
       "      <td>0.83</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>248738</td>\n",
       "      <td>29.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>13</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Bangkok, Thailand / Roseville, IL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>427</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>Hamalainen, Master. Viljo</td>\n",
       "      <td>male</td>\n",
       "      <td>0.67</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>250649</td>\n",
       "      <td>14.5000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Detroit, MI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>548</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>Richards, Master. George Sibley</td>\n",
       "      <td>male</td>\n",
       "      <td>0.83</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>29106</td>\n",
       "      <td>18.7500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Cornwall / Akron, OH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>590</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>West, Miss. Barbara J</td>\n",
       "      <td>female</td>\n",
       "      <td>0.92</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>C.A. 34651</td>\n",
       "      <td>27.7500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>10</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Bournmouth, England</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>611</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Aks, Master. Philip Frank</td>\n",
       "      <td>male</td>\n",
       "      <td>0.83</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>392091</td>\n",
       "      <td>9.3500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>11</td>\n",
       "      <td>NaN</td>\n",
       "      <td>London, England Norfolk, VA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>657</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Baclini, Miss. Eugenie</td>\n",
       "      <td>female</td>\n",
       "      <td>0.75</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2666</td>\n",
       "      <td>19.2583</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "      <td>C</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syria New York, NY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>658</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Baclini, Miss. Helene Barbara</td>\n",
       "      <td>female</td>\n",
       "      <td>0.75</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2666</td>\n",
       "      <td>19.2583</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "      <td>C</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syria New York, NY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>747</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>Danbom, Master. Gilbert Sigvard Emanuel</td>\n",
       "      <td>male</td>\n",
       "      <td>0.33</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>347080</td>\n",
       "      <td>14.4000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Stanton, IA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>763</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Dean, Miss. Elizabeth Gladys \"Millvina\"</td>\n",
       "      <td>female</td>\n",
       "      <td>0.17</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>C.A. 2315</td>\n",
       "      <td>20.5750</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>10</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Devon, England Wichita, KS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1111</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>Peacock, Master. Alfred Edward</td>\n",
       "      <td>male</td>\n",
       "      <td>0.75</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>SOTON/O.Q. 3101315</td>\n",
       "      <td>13.7750</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1240</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Thomas, Master. Assad Alexander</td>\n",
       "      <td>male</td>\n",
       "      <td>0.42</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2625</td>\n",
       "      <td>8.5167</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "      <td>16</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      pclass  survived                                     name     sex   age  \\\n",
       "1          1         1           Allison, Master. Hudson Trevor    male  0.92   \n",
       "359        2         1            Caldwell, Master. Alden Gates    male  0.83   \n",
       "427        2         1                Hamalainen, Master. Viljo    male  0.67   \n",
       "548        2         1          Richards, Master. George Sibley    male  0.83   \n",
       "590        2         1                    West, Miss. Barbara J  female  0.92   \n",
       "611        3         1                Aks, Master. Philip Frank    male  0.83   \n",
       "657        3         1                   Baclini, Miss. Eugenie  female  0.75   \n",
       "658        3         1            Baclini, Miss. Helene Barbara  female  0.75   \n",
       "747        3         0  Danbom, Master. Gilbert Sigvard Emanuel    male  0.33   \n",
       "763        3         1  Dean, Miss. Elizabeth Gladys \"Millvina\"  female  0.17   \n",
       "1111       3         0           Peacock, Master. Alfred Edward    male  0.75   \n",
       "1240       3         1          Thomas, Master. Assad Alexander    male  0.42   \n",
       "\n",
       "      sibsp  parch              ticket      fare    cabin embarked boat  body  \\\n",
       "1         1      2              113781  151.5500  C22 C26        S   11   NaN   \n",
       "359       0      2              248738   29.0000      NaN        S   13   NaN   \n",
       "427       1      1              250649   14.5000      NaN        S    4   NaN   \n",
       "548       1      1               29106   18.7500      NaN        S    4   NaN   \n",
       "590       1      2          C.A. 34651   27.7500      NaN        S   10   NaN   \n",
       "611       0      1              392091    9.3500      NaN        S   11   NaN   \n",
       "657       2      1                2666   19.2583      NaN        C    C   NaN   \n",
       "658       2      1                2666   19.2583      NaN        C    C   NaN   \n",
       "747       0      2              347080   14.4000      NaN        S  NaN   NaN   \n",
       "763       1      2           C.A. 2315   20.5750      NaN        S   10   NaN   \n",
       "1111      1      1  SOTON/O.Q. 3101315   13.7750      NaN        S  NaN   NaN   \n",
       "1240      0      1                2625    8.5167      NaN        C   16   NaN   \n",
       "\n",
       "                              home.dest  \n",
       "1       Montreal, PQ / Chesterville, ON  \n",
       "359   Bangkok, Thailand / Roseville, IL  \n",
       "427                         Detroit, MI  \n",
       "548                Cornwall / Akron, OH  \n",
       "590                 Bournmouth, England  \n",
       "611         London, England Norfolk, VA  \n",
       "657                  Syria New York, NY  \n",
       "658                  Syria New York, NY  \n",
       "747                         Stanton, IA  \n",
       "763          Devon, England Wichita, KS  \n",
       "1111                                NaN  \n",
       "1240                                NaN  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1[df1['age'] < 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07789680",
   "metadata": {},
   "source": [
    "#### What does age = 0.17 mean?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "523c4284",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:44:22.199867Z",
     "start_time": "2022-05-03T08:44:21.635873Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAR1klEQVR4nO3da4xc5X3H8e+/kFLiTcHUYeXYVpdKbhrAjROvHFKqaje0wYEqTqQmMiKRrVA5L4hKWkuVSaUmUWSJF3HSSrmobk1BJWXrAikW5EZdtlGqJgQTp7ZxXNx4RYypnQuXLEU06/z7Yo7LYK89O5fjGT/+fqTRzDznPHN+ux7/9uzZMzORmUiSyvIL/Q4gSeo9y12SCmS5S1KBLHdJKpDlLkkFOr/fAQAWLFiQIyMjHc194YUXmDdvXm8D9YC52mOu9g1qNnO1p5tcO3fu/FFmvnbWhZnZ98uKFSuyUw8//HDHc+tkrvaYq32Dms1c7ekmF/BonqJXPSwjSQWy3CWpQJa7JBXIcpekAlnuklQgy12SCmS5S1KBLHdJKpDlLkkFGoi3H1B7RjY+2PHcDctmWNfF/Knbru94rqQzxz13SSqQ5S5JBbLcJalAlrskFchyl6QCWe6SVCDLXZIKZLlLUoEsd0kqkOUuSQWy3CWpQJa7JBXIcpekAlnuklSgluUeEUsi4uGI2BcReyPilmr8YxHxVETsqi7XNc25NSIORMT+iLi2zi9AknSyubyf+wywITMfi4jXADsj4qFq2acz85PNK0fE5cAa4ArgdcA/R8SvZ+axXgaXJJ1ayz33zHw6Mx+rbv8U2AcsOs2U1cBEZr6UmQeBA8DKXoSVJM1NZObcV44YAb4OXAn8CbAOeB54lMbe/TMR8Rngm5l5VzVnK/DlzLznhMdaD6wHGB4eXjExMdHRFzA9Pc3Q0FBHc+tUZ67dTz3X8dzhC+HIiz0M0yOtci1bdNGZC9NkUJ9fMLjZzNWebnKNj4/vzMzR2ZbN+WP2ImIIuBf4cGY+HxGfBz4BZHW9GfgAELNMP+knSGZuAbYAjI6O5tjY2FyjvMLk5CSdzq1Tnbm6+Zi8Dctm2Lx78D5dsVWuqRvHzlyYJoP6/ILBzWau9tSVa05ny0TEq2gU+xcy8z6AzDySmccy8+fAX/PyoZdDwJKm6YuBw72LLElqZS5nywSwFdiXmZ9qGl/YtNq7gT3V7e3Amoi4ICIuA5YCj/QusiSplbn8fn418H5gd0TsqsY+AtwQEctpHHKZAj4IkJl7I2Ib8DiNM21u9kwZSTqzWpZ7Zn6D2Y+jf+k0czYBm7rIJUnqgq9QlaQCWe6SVCDLXZIKZLlLUoEsd0kqkOUuSQWy3CWpQJa7JBXIcpekAlnuklQgy12SCmS5S1KBLHdJKpDlLkkFstwlqUCWuyQVyHKXpAJZ7pJUIMtdkgpkuUtSgSx3SSqQ5S5JBbLcJalAlrskFchyl6QCWe6SVCDLXZIKZLlLUoFalntELImIhyNiX0TsjYhbqvFLIuKhiHiiup7fNOfWiDgQEfsj4to6vwBJ0snmsuc+A2zIzDcAVwE3R8TlwEZgR2YuBXZU96mWrQGuAFYBn4uI8+oIL0maXctyz8ynM/Ox6vZPgX3AImA1cGe12p3Au6rbq4GJzHwpMw8CB4CVPc4tSTqNto65R8QI8CbgW8BwZj4NjR8AwKXVaouAHzRNO1SNSZLOkMjMua0YMQT8K7ApM++LiGcz8+Km5c9k5vyI+Czw75l5VzW+FfhSZt57wuOtB9YDDA8Pr5iYmOjoC5ienmZoaKijuXWqM9fup57reO7whXDkxR6G6ZFWuZYtuujMhWkyqM8vGNxs5mpPN7nGx8d3ZubobMvOn8sDRMSrgHuBL2TmfdXwkYhYmJlPR8RC4Gg1fghY0jR9MXD4xMfMzC3AFoDR0dEcGxubS5STTE5O0uncOtWZa93GBzueu2HZDJt3z+mf/YxqlWvqxrEzF6bJoD6/YHCzmas9deWay9kyAWwF9mXmp5oWbQfWVrfXAvc3ja+JiAsi4jJgKfBI7yJLklqZyy7c1cD7gd0Rsasa+whwG7AtIm4CngTeA5CZeyNiG/A4jTNtbs7MY70OLkk6tZblnpnfAOIUi685xZxNwKYuckmSuuArVCWpQJa7JBXIcpekAlnuklQgy12SCmS5S1KBLHdJKpDlLkkFstwlqUCWuyQVyHKXpAJZ7pJUIMtdkgpkuUtSgSx3SSqQ5S5JBbLcJalAlrskFchyl6QCWe6SVCDLXZIKZLlLUoEsd0kqkOUuSQU6v98BpLkY2fhgX7a7YdkMY33ZstQd99wlqUCWuyQVyHKXpAJZ7pJUoJblHhG3R8TRiNjTNPaxiHgqInZVl+ualt0aEQciYn9EXFtXcEnSqc1lz/0OYNUs45/OzOXV5UsAEXE5sAa4oprzuYg4r1dhJUlz07LcM/PrwE/m+HirgYnMfCkzDwIHgJVd5JMkdSAys/VKESPAA5l5ZXX/Y8A64HngUWBDZj4TEZ8BvpmZd1XrbQW+nJn3zPKY64H1AMPDwysmJiY6+gKmp6cZGhrqaG6d6sy1+6nnOp47fCEcebGHYXpkkHNdeslF/Y4xq3Pxud+NEnONj4/vzMzR2ZZ1+iKmzwOfALK63gx8AIhZ1p31p0dmbgG2AIyOjubY2FhHQSYnJ+l0bp3qzLWuixf0bFg2w+bdg/fatUHO9d4BfH7Bufnc78a5lqujs2Uy80hmHsvMnwN/zcuHXg4BS5pWXQwc7i6iJKldHZV7RCxsuvtu4PiZNNuBNRFxQURcBiwFHukuoiSpXS1/D46Iu4ExYEFEHAI+CoxFxHIah1ymgA8CZObeiNgGPA7MADdn5rFakkuSTqlluWfmDbMMbz3N+puATd2EkiR1x1eoSlKBLHdJKpDlLkkFstwlqUCWuyQVyHKXpAJZ7pJUIMtdkgpkuUtSgSx3SSqQ5S5JBbLcJalAlrskFchyl6QCWe6SVCDLXZIKZLlLUoEsd0kqkOUuSQWy3CWpQJa7JBXIcpekAlnuklQgy12SCmS5S1KBLHdJKtD5/Q4gDbqRjQ/2ZbtTt13fl+2qDO65S1KBLHdJKlDLco+I2yPiaETsaRq7JCIeiognquv5TctujYgDEbE/Iq6tK7gk6dTmsud+B7DqhLGNwI7MXArsqO4TEZcDa4Arqjmfi4jzepZWkjQnLcs9M78O/OSE4dXAndXtO4F3NY1PZOZLmXkQOACs7E1USdJcRWa2XiliBHggM6+s7j+bmRc3LX8mM+dHxGeAb2bmXdX4VuDLmXnPLI+5HlgPMDw8vGJiYqKjL2B6epqhoaGO5tapzly7n3qu47nDF8KRF3sYpkfMdbJliy467fJz8bnfjRJzjY+P78zM0dmW9fpUyJhlbNafHpm5BdgCMDo6mmNjYx1tcHJykk7n1qnOXOu6ODVvw7IZNu8evDNgzXWyqRvHTrv8XHzud+Ncy9Xp2TJHImIhQHV9tBo/BCxpWm8xcLjzeJKkTnRa7tuBtdXttcD9TeNrIuKCiLgMWAo80l1ESVK7Wv6+GRF3A2PAgog4BHwUuA3YFhE3AU8C7wHIzL0RsQ14HJgBbs7MYzVllySdQstyz8wbTrHomlOsvwnY1E0oSVJ3fIWqJBXIcpekAlnuklQgy12SCmS5S1KBLHdJKpDlLkkFstwlqUCWuyQVyHKXpAJZ7pJUIMtdkgpkuUtSgSx3SSqQ5S5JBRq8D63swEgXnynajanbru/LdiWpFffcJalAlrskFchyl6QCWe6SVCDLXZIKVMTZMlKJWp0FtmHZDOtqOlPMM8HOfu65S1KB3HPvwun2rOrcq5KkVtxzl6QCWe6SVCDLXZIKZLlLUoEsd0kqUFdny0TEFPBT4Bgwk5mjEXEJ8A/ACDAFvDczn+kupiSpHb3Ycx/PzOWZOVrd3wjsyMylwI7qviTpDKrjsMxq4M7q9p3Au2rYhiTpNCIzO58ccRB4BkjgrzJzS0Q8m5kXN63zTGbOn2XuemA9wPDw8IqJiYmOMkxPT3PwuWMdza3T8IVw5MV+pziZudozqLmg3mzLFl3U8dzp6WmGhoZ6mKY3Ssw1Pj6+s+moySt0+wrVqzPzcERcCjwUEd+b68TM3AJsARgdHc2xsbGOAkxOTrL5Gy90NLdOG5bNsHn34L0A2FztGdRcUG+2qRvHOp47OTlJp/+f63Su5erqsExmHq6ujwJfBFYCRyJiIUB1fbTbkJKk9nRc7hExLyJec/w28HZgD7AdWFuttha4v9uQkqT2dPM73TDwxYg4/jh/n5lfiYhvA9si4ibgSeA93ceUJLWj43LPzO8Db5xl/MfANd2EkiR1x1eoSlKBBvM0AEl91epToE6nm88y8BOgesc9d0kqkOUuSQWy3CWpQJa7JBXIcpekAlnuklQgy12SCmS5S1KBLHdJKpDlLkkFstwlqUCWuyQVyHKXpAJZ7pJUIMtdkgpkuUtSgSx3SSqQ5S5JBbLcJalAlrskFchyl6QCWe6SVKDz+x1Ako4b2fhgbY+9YdkM607x+FO3XV/bdvvFPXdJKpDlLkkFstwlqUCWuyQVqLZyj4hVEbE/Ig5ExMa6tiNJOlktZ8tExHnAZ4HfAw4B346I7Zn5eB3bk6Ru1HmWTit3rJpXy+PWtee+EjiQmd/PzP8FJoDVNW1LknSCyMzeP2jEHwCrMvMPq/vvB96SmR9qWmc9sL66+3pgf4ebWwD8qIu4dTFXe8zVvkHNZq72dJPrVzPztbMtqOtFTDHL2Ct+imTmFmBL1xuKeDQzR7t9nF4zV3vM1b5BzWau9tSVq67DMoeAJU33FwOHa9qWJOkEdZX7t4GlEXFZRPwisAbYXtO2JEknqOWwTGbORMSHgK8C5wG3Z+beOrZFDw7t1MRc7TFX+wY1m7naU0uuWv6gKknqL1+hKkkFstwlqUBnbbkP0tsbRMTtEXE0IvY0jV0SEQ9FxBPV9fw+5FoSEQ9HxL6I2BsRtwxCtoj4pYh4JCK+W+X6+CDkqjKcFxHfiYgHBiVTlWMqInZHxK6IeHRQskXExRFxT0R8r3qevbXfuSLi9dX36fjl+Yj4cL9zVdn+uHrO74mIu6v/C7XkOivLventDd4BXA7cEBGX9zHSHcCqE8Y2Ajsycymwo7p/ps0AGzLzDcBVwM3V96nf2V4C3paZbwSWA6si4qoByAVwC7Cv6f4gZDpuPDOXN50TPQjZ/hL4Smb+BvBGGt+7vubKzP3V92k5sAL4H+CL/c4VEYuAPwJGM/NKGiebrKktV2aedRfgrcBXm+7fCtza50wjwJ6m+/uBhdXthcD+Afi+3U/j/X4GJhvwauAx4C39zkXj9Rg7gLcBDwzSvyMwBSw4Yazf369fBg5SnZgxKLlOyPJ24N8GIRewCPgBcAmNMxUfqPLVkuus3HPn5W/ScYeqsUEynJlPA1TXl/YzTESMAG8CvsUAZKsOf+wCjgIPZeYg5PoL4E+BnzeN9TvTcQl8LSJ2Vm/dMQjZfg34IfC31aGsv4mIeQOQq9ka4O7qdl9zZeZTwCeBJ4Gngecy82t15Tpby73l2xvoZRExBNwLfDgzn+93HoDMPJaNX5sXAysj4sp+5omI3weOZubOfuY4jasz8800DkXeHBG/0+9ANPY+3wx8PjPfBLxAfw9bvUL1Asp3Av/Y7ywA1bH01cBlwOuAeRHxvrq2d7aW+9nw9gZHImIhQHV9tB8hIuJVNIr9C5l53yBlA8jMZ4FJGn+z6Geuq4F3RsQUjXcxfVtE3NXnTP8vMw9X10dpHD9eOQDZDgGHqt+6AO6hUfb9znXcO4DHMvNIdb/fuX4XOJiZP8zMnwH3Ab9VV66ztdzPhrc32A6srW6vpXG8+4yKiAC2Avsy81ODki0iXhsRF1e3L6TxpP9eP3Nl5q2ZuTgzR2g8n/4lM9/Xz0zHRcS8iHjN8ds0jtPu6Xe2zPxv4AcR8fpq6Brg8X7nanIDLx+Sgf7nehK4KiJeXf3fvIbGH6DrydWvP3T04I8T1wH/CfwX8Gd9znI3jWNoP6OxN3MT8Cs0/jj3RHV9SR9y/TaNw1X/AeyqLtf1Oxvwm8B3qlx7gD+vxvv+PatyjPHyH1T7nonGse3vVpe9x5/vA5JtOfBo9W/5T8D8Acn1auDHwEVNY4OQ6+M0dmT2AH8HXFBXLt9+QJIKdLYelpEknYblLkkFstwlqUCWuyQVyHKXpAJZ7pJUIMtdkgr0fw1Hs4Oa6xggAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "df1['age'].hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b4c2999d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:44:34.640973Z",
     "start_time": "2022-05-03T08:44:34.629391Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "263"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(df1['age'].isnull())\n",
    "# 263 passengers have no age recorded."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b224f0b",
   "metadata": {},
   "source": [
    "### Let's look at other factors, e.g. names can give a clue because most passengers seem to have a title (e.g. 'Mrs')\n",
    "\n",
    "### Discussion:\n",
    "\n",
    "* Can we assume that 'Mrs' implies married (implies not a child)?\n",
    "\n",
    "* What about 'Miss' or 'Ms'?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "3622dcee",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:44:59.387968Z",
     "start_time": "2022-05-03T08:44:59.383381Z"
    }
   },
   "outputs": [],
   "source": [
    "# Look into titles, e.g. 'Mrs' implies married (implies not child)\n",
    "def name_extract(word):\n",
    "     return word.split(',')[1].split('.')[0].strip()\n",
    "    \n",
    "# because names are in this format:\n",
    "# Allison, Master. Hudson Trevor\n",
    "# we can split on ','\n",
    "# then '.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ca60f583",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:45:07.193182Z",
     "start_time": "2022-05-03T08:45:07.183507Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Miss', 'Master', 'Mr', 'Mrs', 'Col', 'Mme', 'Dr', 'Major', 'Capt',\n",
       "       'Lady', 'Sir', 'Mlle', 'Dona', 'Jonkheer', 'the Countess', 'Don',\n",
       "       'Rev', 'Ms'], dtype=object)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "temp = pd.DataFrame({'Title':df1['name'].apply(name_extract)}) # testing, apply the method to the data\n",
    "# check unique values\n",
    "temp['Title'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "8e775684",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:45:20.706465Z",
     "start_time": "2022-05-03T08:45:20.698198Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Mr              757\n",
       "Miss            260\n",
       "Mrs             197\n",
       "Master           61\n",
       "Rev               8\n",
       "Dr                8\n",
       "Col               4\n",
       "Mlle              2\n",
       "Ms                2\n",
       "Major             2\n",
       "Capt              1\n",
       "Sir               1\n",
       "Dona              1\n",
       "Jonkheer          1\n",
       "the Countess      1\n",
       "Don               1\n",
       "Mme               1\n",
       "Lady              1\n",
       "Name: Title, dtype: int64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# a couple of strange ones but most of the standard titles are there\n",
    "temp['Title'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "437fc98a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:45:31.743320Z",
     "start_time": "2022-05-03T08:45:31.733280Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1309"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# did we miss any?\n",
    "sum(temp['Title'].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c40bedce",
   "metadata": {},
   "source": [
    "### That worked really well, we have titles for everyone and there are only a few strange ones:\n",
    "\n",
    "Someone look up 'Jonkheer' (and Dona - Spanish?)\n",
    "\n",
    "#### So what can we say about 'Master'? Usually refers to boys?\n",
    "\n",
    "#### Dr, Rev, Col, Major, Capt, Sir, Don, would all be adults? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c7d21777",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:46:17.698844Z",
     "start_time": "2022-05-03T08:46:17.678059Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pclass</th>\n",
       "      <th>survived</th>\n",
       "      <th>name</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>ticket</th>\n",
       "      <th>fare</th>\n",
       "      <th>cabin</th>\n",
       "      <th>embarked</th>\n",
       "      <th>boat</th>\n",
       "      <th>body</th>\n",
       "      <th>home.dest</th>\n",
       "      <th>Title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Allen, Miss. Elisabeth Walton</td>\n",
       "      <td>female</td>\n",
       "      <td>29.00</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>24160</td>\n",
       "      <td>211.3375</td>\n",
       "      <td>B5</td>\n",
       "      <td>S</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>St Louis, MO</td>\n",
       "      <td>Miss</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Allison, Master. Hudson Trevor</td>\n",
       "      <td>male</td>\n",
       "      <td>0.92</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>113781</td>\n",
       "      <td>151.5500</td>\n",
       "      <td>C22 C26</td>\n",
       "      <td>S</td>\n",
       "      <td>11</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Montreal, PQ / Chesterville, ON</td>\n",
       "      <td>Master</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Allison, Miss. Helen Loraine</td>\n",
       "      <td>female</td>\n",
       "      <td>2.00</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>113781</td>\n",
       "      <td>151.5500</td>\n",
       "      <td>C22 C26</td>\n",
       "      <td>S</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Montreal, PQ / Chesterville, ON</td>\n",
       "      <td>Miss</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Allison, Mr. Hudson Joshua Creighton</td>\n",
       "      <td>male</td>\n",
       "      <td>30.00</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>113781</td>\n",
       "      <td>151.5500</td>\n",
       "      <td>C22 C26</td>\n",
       "      <td>S</td>\n",
       "      <td>NaN</td>\n",
       "      <td>135.0</td>\n",
       "      <td>Montreal, PQ / Chesterville, ON</td>\n",
       "      <td>Mr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Allison, Mrs. Hudson J C (Bessie Waldo Daniels)</td>\n",
       "      <td>female</td>\n",
       "      <td>25.00</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>113781</td>\n",
       "      <td>151.5500</td>\n",
       "      <td>C22 C26</td>\n",
       "      <td>S</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Montreal, PQ / Chesterville, ON</td>\n",
       "      <td>Mrs</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   pclass  survived                                             name     sex  \\\n",
       "0       1         1                    Allen, Miss. Elisabeth Walton  female   \n",
       "1       1         1                   Allison, Master. Hudson Trevor    male   \n",
       "2       1         0                     Allison, Miss. Helen Loraine  female   \n",
       "3       1         0             Allison, Mr. Hudson Joshua Creighton    male   \n",
       "4       1         0  Allison, Mrs. Hudson J C (Bessie Waldo Daniels)  female   \n",
       "\n",
       "     age  sibsp  parch  ticket      fare    cabin embarked boat   body  \\\n",
       "0  29.00      0      0   24160  211.3375       B5        S    2    NaN   \n",
       "1   0.92      1      2  113781  151.5500  C22 C26        S   11    NaN   \n",
       "2   2.00      1      2  113781  151.5500  C22 C26        S  NaN    NaN   \n",
       "3  30.00      1      2  113781  151.5500  C22 C26        S  NaN  135.0   \n",
       "4  25.00      1      2  113781  151.5500  C22 C26        S  NaN    NaN   \n",
       "\n",
       "                         home.dest   Title  \n",
       "0                     St Louis, MO    Miss  \n",
       "1  Montreal, PQ / Chesterville, ON  Master  \n",
       "2  Montreal, PQ / Chesterville, ON    Miss  \n",
       "3  Montreal, PQ / Chesterville, ON      Mr  \n",
       "4  Montreal, PQ / Chesterville, ON     Mrs  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = df1 # copy then insert new column\n",
    "df2['Title'] = df1['name'].apply(name_extract)\n",
    "df2.head() # title at far right"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "7782f5b2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:46:34.505873Z",
     "start_time": "2022-05-03T08:46:34.492697Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pclass</th>\n",
       "      <th>survived</th>\n",
       "      <th>name</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>ticket</th>\n",
       "      <th>fare</th>\n",
       "      <th>cabin</th>\n",
       "      <th>embarked</th>\n",
       "      <th>boat</th>\n",
       "      <th>body</th>\n",
       "      <th>home.dest</th>\n",
       "      <th>Title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [pclass, survived, name, sex, age, sibsp, parch, ticket, fare, cabin, embarked, boat, body, home.dest, Title]\n",
       "Index: []"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# just check (again) we got most of them\n",
    "df2[df2['Title'].isnull()]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0fe3704",
   "metadata": {},
   "source": [
    "#### First let's check distribution of missing values across different titles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "bd779fa6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:46:51.749852Z",
     "start_time": "2022-05-03T08:46:51.739429Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Mr        176\n",
       "Miss       50\n",
       "Mrs        27\n",
       "Master      8\n",
       "Dr          1\n",
       "Ms          1\n",
       "Name: Title, dtype: int64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2[df2.age.isnull()].Title.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d52b168",
   "metadata": {},
   "source": [
    "### We will first investigate missing values in doctors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "cf07b6c0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:47:08.482850Z",
     "start_time": "2022-05-03T08:47:08.462539Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pclass</th>\n",
       "      <th>survived</th>\n",
       "      <th>name</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>ticket</th>\n",
       "      <th>fare</th>\n",
       "      <th>cabin</th>\n",
       "      <th>embarked</th>\n",
       "      <th>boat</th>\n",
       "      <th>body</th>\n",
       "      <th>home.dest</th>\n",
       "      <th>Title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Brewe, Dr. Arthur Jackson</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>112379</td>\n",
       "      <td>39.6000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Philadelphia, PA</td>\n",
       "      <td>Dr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Dodge, Dr. Washington</td>\n",
       "      <td>male</td>\n",
       "      <td>53.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>33638</td>\n",
       "      <td>81.8583</td>\n",
       "      <td>A34</td>\n",
       "      <td>S</td>\n",
       "      <td>13</td>\n",
       "      <td>NaN</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>Dr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>119</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Frauenthal, Dr. Henry William</td>\n",
       "      <td>male</td>\n",
       "      <td>50.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17611</td>\n",
       "      <td>133.6500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>New York, NY</td>\n",
       "      <td>Dr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>181</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Leader, Dr. Alice (Farnham)</td>\n",
       "      <td>female</td>\n",
       "      <td>49.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>17465</td>\n",
       "      <td>25.9292</td>\n",
       "      <td>D17</td>\n",
       "      <td>S</td>\n",
       "      <td>8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>New York, NY</td>\n",
       "      <td>Dr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>206</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Minahan, Dr. William Edward</td>\n",
       "      <td>male</td>\n",
       "      <td>44.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>19928</td>\n",
       "      <td>90.0000</td>\n",
       "      <td>C78</td>\n",
       "      <td>Q</td>\n",
       "      <td>NaN</td>\n",
       "      <td>230.0</td>\n",
       "      <td>Fond du Lac, WI</td>\n",
       "      <td>Dr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>278</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Stahelin-Maeglin, Dr. Max</td>\n",
       "      <td>male</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13214</td>\n",
       "      <td>30.5000</td>\n",
       "      <td>B50</td>\n",
       "      <td>C</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Basel, Switzerland</td>\n",
       "      <td>Dr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>508</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>Moraweck, Dr. Ernest</td>\n",
       "      <td>male</td>\n",
       "      <td>54.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>29011</td>\n",
       "      <td>14.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Frankfort, KY</td>\n",
       "      <td>Dr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>525</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>Pain, Dr. Alfred</td>\n",
       "      <td>male</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>244278</td>\n",
       "      <td>10.5000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Hamilton, ON</td>\n",
       "      <td>Dr</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     pclass  survived                           name     sex   age  sibsp  \\\n",
       "40        1         0      Brewe, Dr. Arthur Jackson    male   NaN      0   \n",
       "93        1         1          Dodge, Dr. Washington    male  53.0      1   \n",
       "119       1         1  Frauenthal, Dr. Henry William    male  50.0      2   \n",
       "181       1         1    Leader, Dr. Alice (Farnham)  female  49.0      0   \n",
       "206       1         0    Minahan, Dr. William Edward    male  44.0      2   \n",
       "278       1         1      Stahelin-Maeglin, Dr. Max    male  32.0      0   \n",
       "508       2         0           Moraweck, Dr. Ernest    male  54.0      0   \n",
       "525       2         0               Pain, Dr. Alfred    male  23.0      0   \n",
       "\n",
       "     parch    ticket      fare cabin embarked boat   body           home.dest  \\\n",
       "40       0    112379   39.6000   NaN        C  NaN    NaN    Philadelphia, PA   \n",
       "93       1     33638   81.8583   A34        S   13    NaN   San Francisco, CA   \n",
       "119      0  PC 17611  133.6500   NaN        S    5    NaN        New York, NY   \n",
       "181      0     17465   25.9292   D17        S    8    NaN        New York, NY   \n",
       "206      0     19928   90.0000   C78        Q  NaN  230.0     Fond du Lac, WI   \n",
       "278      0     13214   30.5000   B50        C    3    NaN  Basel, Switzerland   \n",
       "508      0     29011   14.0000   NaN        S  NaN    NaN       Frankfort, KY   \n",
       "525      0    244278   10.5000   NaN        S  NaN    NaN        Hamilton, ON   \n",
       "\n",
       "    Title  \n",
       "40     Dr  \n",
       "93     Dr  \n",
       "119    Dr  \n",
       "181    Dr  \n",
       "206    Dr  \n",
       "278    Dr  \n",
       "508    Dr  \n",
       "525    Dr  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2[df2['Title'] == \"Dr\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27779acb",
   "metadata": {},
   "source": [
    "Drs are mostly older (except for Dr Pain - hah)\n",
    "\n",
    "#### What's the average age of Drs?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "6294104f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:47:29.688534Z",
     "start_time": "2022-05-03T08:47:29.675084Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/4n/y3z900hx7yj7qq3g3q2t8y7m0000gn/T/ipykernel_11728/2031813476.py:1: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError.  Select only valid columns before calling the reduction.\n",
      "  df2[df2['Title'] == \"Dr\"].mean()\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "pclass        1.250000\n",
       "survived      0.500000\n",
       "age          43.571429\n",
       "sibsp         0.625000\n",
       "parch         0.125000\n",
       "fare         53.254688\n",
       "body        230.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2[df2['Title'] == \"Dr\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ed964f1",
   "metadata": {},
   "source": [
    "The average age for all passengers is 29, for Drs it's 43, an improvement in fidelity?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "7f8827bf",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:47:49.771250Z",
     "start_time": "2022-05-03T08:47:49.759249Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/4n/y3z900hx7yj7qq3g3q2t8y7m0000gn/T/ipykernel_11728/4008260498.py:1: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError.  Select only valid columns before calling the reduction.\n",
      "  df2[(df2['Title'] == \"Dr\") & (df2['sex'] == \"male\")].mean()\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "pclass        1.285714\n",
       "survived      0.428571\n",
       "age          42.666667\n",
       "sibsp         0.714286\n",
       "parch         0.142857\n",
       "fare         57.158329\n",
       "body        230.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2[(df2['Title'] == \"Dr\") & (df2['sex'] == \"male\")].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1946a79a",
   "metadata": {},
   "source": [
    "no missing values so move on."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36a7f9c8",
   "metadata": {},
   "source": [
    "## Missing values: ages for boys\n",
    "\n",
    "It seems, in this era (pre Jackie Chan), Master did mean boys "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "d3ad5296",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:48:27.819270Z",
     "start_time": "2022-05-03T08:48:27.792740Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pclass</th>\n",
       "      <th>survived</th>\n",
       "      <th>name</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>ticket</th>\n",
       "      <th>fare</th>\n",
       "      <th>cabin</th>\n",
       "      <th>embarked</th>\n",
       "      <th>boat</th>\n",
       "      <th>body</th>\n",
       "      <th>home.dest</th>\n",
       "      <th>Title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Allison, Master. Hudson Trevor</td>\n",
       "      <td>male</td>\n",
       "      <td>0.92</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>113781</td>\n",
       "      <td>151.5500</td>\n",
       "      <td>C22 C26</td>\n",
       "      <td>S</td>\n",
       "      <td>11</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Montreal, PQ / Chesterville, ON</td>\n",
       "      <td>Master</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Carter, Master. William Thornton II</td>\n",
       "      <td>male</td>\n",
       "      <td>11.00</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>113760</td>\n",
       "      <td>120.0000</td>\n",
       "      <td>B96 B98</td>\n",
       "      <td>S</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Bryn Mawr, PA</td>\n",
       "      <td>Master</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Dodge, Master. Washington</td>\n",
       "      <td>male</td>\n",
       "      <td>4.00</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>33638</td>\n",
       "      <td>81.8583</td>\n",
       "      <td>A34</td>\n",
       "      <td>S</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>Master</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>249</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Ryerson, Master. John Borie</td>\n",
       "      <td>male</td>\n",
       "      <td>13.00</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>PC 17608</td>\n",
       "      <td>262.3750</td>\n",
       "      <td>B57 B59 B63 B66</td>\n",
       "      <td>C</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Haverford, PA / Cooperstown, NY</td>\n",
       "      <td>Master</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>273</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Spedden, Master. Robert Douglas</td>\n",
       "      <td>male</td>\n",
       "      <td>6.00</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>16966</td>\n",
       "      <td>134.5000</td>\n",
       "      <td>E34</td>\n",
       "      <td>C</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Tuxedo Park, NY</td>\n",
       "      <td>Master</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1207</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>Skoog, Master. Karl Thorsten</td>\n",
       "      <td>male</td>\n",
       "      <td>10.00</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>347088</td>\n",
       "      <td>27.9000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Master</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1240</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Thomas, Master. Assad Alexander</td>\n",
       "      <td>male</td>\n",
       "      <td>0.42</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2625</td>\n",
       "      <td>8.5167</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "      <td>16</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Master</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1256</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Touma, Master. Georges Youssef</td>\n",
       "      <td>male</td>\n",
       "      <td>7.00</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2650</td>\n",
       "      <td>15.2458</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "      <td>C</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Master</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1262</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>van Billiard, Master. James William</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>A/5. 851</td>\n",
       "      <td>14.5000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Master</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1263</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>van Billiard, Master. Walter John</td>\n",
       "      <td>male</td>\n",
       "      <td>11.50</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>A/5. 851</td>\n",
       "      <td>14.5000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Master</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>61 rows × 15 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      pclass  survived                                 name   sex    age  \\\n",
       "1          1         1       Allison, Master. Hudson Trevor  male   0.92   \n",
       "54         1         1  Carter, Master. William Thornton II  male  11.00   \n",
       "94         1         1            Dodge, Master. Washington  male   4.00   \n",
       "249        1         1          Ryerson, Master. John Borie  male  13.00   \n",
       "273        1         1      Spedden, Master. Robert Douglas  male   6.00   \n",
       "...      ...       ...                                  ...   ...    ...   \n",
       "1207       3         0         Skoog, Master. Karl Thorsten  male  10.00   \n",
       "1240       3         1      Thomas, Master. Assad Alexander  male   0.42   \n",
       "1256       3         1       Touma, Master. Georges Youssef  male   7.00   \n",
       "1262       3         0  van Billiard, Master. James William  male    NaN   \n",
       "1263       3         0    van Billiard, Master. Walter John  male  11.50   \n",
       "\n",
       "      sibsp  parch    ticket      fare            cabin embarked boat  body  \\\n",
       "1         1      2    113781  151.5500          C22 C26        S   11   NaN   \n",
       "54        1      2    113760  120.0000          B96 B98        S    4   NaN   \n",
       "94        0      2     33638   81.8583              A34        S    5   NaN   \n",
       "249       2      2  PC 17608  262.3750  B57 B59 B63 B66        C    4   NaN   \n",
       "273       0      2     16966  134.5000              E34        C    3   NaN   \n",
       "...     ...    ...       ...       ...              ...      ...  ...   ...   \n",
       "1207      3      2    347088   27.9000              NaN        S  NaN   NaN   \n",
       "1240      0      1      2625    8.5167              NaN        C   16   NaN   \n",
       "1256      1      1      2650   15.2458              NaN        C    C   NaN   \n",
       "1262      1      1  A/5. 851   14.5000              NaN        S  NaN   NaN   \n",
       "1263      1      1  A/5. 851   14.5000              NaN        S  NaN   1.0   \n",
       "\n",
       "                            home.dest   Title  \n",
       "1     Montreal, PQ / Chesterville, ON  Master  \n",
       "54                      Bryn Mawr, PA  Master  \n",
       "94                  San Francisco, CA  Master  \n",
       "249   Haverford, PA / Cooperstown, NY  Master  \n",
       "273                   Tuxedo Park, NY  Master  \n",
       "...                               ...     ...  \n",
       "1207                              NaN  Master  \n",
       "1240                              NaN  Master  \n",
       "1256                              NaN  Master  \n",
       "1262                              NaN  Master  \n",
       "1263                              NaN  Master  \n",
       "\n",
       "[61 rows x 15 columns]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2[df2['Title'] == \"Master\"] # how many?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "5ad0726b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:48:35.868003Z",
     "start_time": "2022-05-03T08:48:35.838652Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pclass</th>\n",
       "      <th>survived</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>body</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>61.000000</td>\n",
       "      <td>61.000000</td>\n",
       "      <td>53.000000</td>\n",
       "      <td>61.000000</td>\n",
       "      <td>61.000000</td>\n",
       "      <td>61.000000</td>\n",
       "      <td>3.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>2.655738</td>\n",
       "      <td>0.508197</td>\n",
       "      <td>5.482642</td>\n",
       "      <td>2.049180</td>\n",
       "      <td>1.377049</td>\n",
       "      <td>36.008195</td>\n",
       "      <td>24.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.629424</td>\n",
       "      <td>0.504082</td>\n",
       "      <td>4.161554</td>\n",
       "      <td>1.961515</td>\n",
       "      <td>0.552688</td>\n",
       "      <td>40.677102</td>\n",
       "      <td>37.26929</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.330000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.170800</td>\n",
       "      <td>1.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>15.245800</td>\n",
       "      <td>2.50000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>26.000000</td>\n",
       "      <td>4.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>36.750000</td>\n",
       "      <td>35.50000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>14.500000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>262.375000</td>\n",
       "      <td>67.00000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          pclass   survived        age      sibsp      parch        fare  \\\n",
       "count  61.000000  61.000000  53.000000  61.000000  61.000000   61.000000   \n",
       "mean    2.655738   0.508197   5.482642   2.049180   1.377049   36.008195   \n",
       "std     0.629424   0.504082   4.161554   1.961515   0.552688   40.677102   \n",
       "min     1.000000   0.000000   0.330000   0.000000   0.000000    3.170800   \n",
       "25%     2.000000   0.000000   2.000000   1.000000   1.000000   15.245800   \n",
       "50%     3.000000   1.000000   4.000000   1.000000   1.000000   26.000000   \n",
       "75%     3.000000   1.000000   9.000000   4.000000   2.000000   36.750000   \n",
       "max     3.000000   1.000000  14.500000   8.000000   2.000000  262.375000   \n",
       "\n",
       "           body  \n",
       "count   3.00000  \n",
       "mean   24.00000  \n",
       "std    37.26929  \n",
       "min     1.00000  \n",
       "25%     2.50000  \n",
       "50%     4.00000  \n",
       "75%    35.50000  \n",
       "max    67.00000  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#there are a lot, 61, use describe()\n",
    "df2[df2['Title'] == \"Master\"].describe() # min age is 0.33 (4 months?), max is 14.5, mean is 5.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "7a0fd89e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:48:42.401823Z",
     "start_time": "2022-05-03T08:48:42.389786Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/4n/y3z900hx7yj7qq3g3q2t8y7m0000gn/T/ipykernel_11728/3018166891.py:1: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError.  Select only valid columns before calling the reduction.\n",
      "  df2[df2['Title'] == \"Master\"].mean()\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "pclass       2.655738\n",
       "survived     0.508197\n",
       "age          5.482642\n",
       "sibsp        2.049180\n",
       "parch        1.377049\n",
       "fare        36.008195\n",
       "body        24.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2[df2['Title'] == \"Master\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be4dd17f",
   "metadata": {},
   "source": [
    "## Mean imputation of age by title \n",
    "\n",
    "In this task you need to calculate the mean age for each title group and use the imputed values to replace the missing values in the corresponding group. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "4857ec04",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T08:49:10.645689Z",
     "start_time": "2022-05-03T08:49:10.631750Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    1309.000000\n",
       "mean       29.896894\n",
       "std        13.193803\n",
       "min         0.170000\n",
       "25%        21.774238\n",
       "50%        30.000000\n",
       "75%        36.000000\n",
       "max        80.000000\n",
       "Name: age, dtype: float64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# this seems too easy, is it right? \n",
    "df2[\"age\"].fillna(df2.groupby(\"Title\")[\"age\"].transform(\"mean\"), inplace=True)\n",
    "df2.age.describe()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}