DW06_Data_Auditing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a1989244",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Data-cleansing-process:\" data-toc-modified-id=\"Data-cleansing-process:-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Data cleansing process:</a></span></li><li><span><a href=\"#Wrangling-Titanic-Data\" data-toc-modified-id=\"Wrangling-Titanic-Data-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Wrangling Titanic Data</a></span><ul class=\"toc-item\"><li><span><a href=\"#First,-load-the-data-using-Pandas-library\" data-toc-modified-id=\"First,-load-the-data-using-Pandas-library-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>First, load the data using Pandas library</a></span></li><li><span><a href=\"#Identify-Syntactical-Anomalies\" data-toc-modified-id=\"Identify-Syntactical-Anomalies-2.2\"><span class=\"toc-item-num\">2.2&nbsp;&nbsp;</span>Identify Syntactical Anomalies</a></span><ul class=\"toc-item\"><li><span><a href=\"#Are-all-the-titles-consistent?\" data-toc-modified-id=\"Are-all-the-titles-consistent?-2.2.1\"><span class=\"toc-item-num\">2.2.1&nbsp;&nbsp;</span>Are all the titles consistent?</a></span></li><li><span><a href=\"#Are-there-any-lexical-errors-in-the-data?\" data-toc-modified-id=\"Are-there-any-lexical-errors-in-the-data?-2.2.2\"><span class=\"toc-item-num\">2.2.2&nbsp;&nbsp;</span>Are there any lexical errors in the data?</a></span></li><li><span><a href=\"#Further-more,-are-their-any-other-inconsistent-spellings?\" data-toc-modified-id=\"Further-more,-are-their-any-other-inconsistent-spellings?-2.2.3\"><span class=\"toc-item-num\">2.2.3&nbsp;&nbsp;</span>Further more, are their any other inconsistent spellings?</a></span></li></ul></li><li><span><a href=\"#Semantic-errors:\" data-toc-modified-id=\"Semantic-errors:-2.3\"><span class=\"toc-item-num\">2.3&nbsp;&nbsp;</span>Semantic errors:</a></span><ul class=\"toc-item\"><li><span><a href=\"#Are-the-values-of-&quot;adult_male&quot;-consistent-with-the-values-of-&quot;sex&quot;-and-&quot;who&quot;?\" data-toc-modified-id=\"Are-the-values-of-&quot;adult_male&quot;-consistent-with-the-values-of-&quot;sex&quot;-and-&quot;who&quot;?-2.3.1\"><span class=\"toc-item-num\">2.3.1&nbsp;&nbsp;</span>Are the values of \"adult_male\" consistent with the values of \"sex\" and \"who\"?</a></span></li><li><span><a href=\"#Are-there-any-duplicated-records?\" data-toc-modified-id=\"Are-there-any-duplicated-records?-2.3.2\"><span class=\"toc-item-num\">2.3.2&nbsp;&nbsp;</span>Are there any duplicated records?</a></span></li></ul></li></ul></li><li><span><a href=\"#Summary\" data-toc-modified-id=\"Summary-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Summary</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f232c37",
   "metadata": {},
   "source": [
    "# Data Auditing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dda1f5bc",
   "metadata": {},
   "source": [
    "## Data cleansing process:\n",
    "\n",
    "Data cleansing is an iterative process. The first step of the cleansing process is data auditing. In this step, we identify the types of anomalies that reduce the data quality.  Data auditing is about programmatically checking the data using some validation rules that are pre-specified, and then creating a report of the quality of the data and its problems. We often apply some statistical tests in this step for examining the data.\n",
    "Data Anomalies can be classified at a high level into three categories:\n",
    "\n",
    "1. **Syntactic Anomalies**: \n",
    "describe characteristics concerning the format and values used for representation of the entities. Syntactic anomalies such as: lexical errors, domain format errors, syntactical error and irregularities.\n",
    "2. **Semantic Anomalies**: \n",
    "hinder the data collection from being a comprehensive and non-redundant representation of the mini-world. These types of anomalies include: Integrity constraint violations, contradictions, duplicates and invalid tuples\n",
    "3. **Coverage Anomalies**: \n",
    "decrease the amount of entities and entity properties from the mini-world that are represented in the data collection. Coverage anomalies are categorized as: missing values and missing tuples\n",
    "\n",
    "We give examples in this part of the auditing process that is applied to discover different anomalies in data.\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64553681",
   "metadata": {},
   "source": [
    "## Wrangling Titanic Data\n",
    "\n",
    "The Titanic data is the data set provided in the Kaggle competition \"Titanic: Machine Learning from Disaster\". The competition has been available from 28 Sep 2012 with more than 4000 teams joining the competition. \n",
    "\n",
    "\"The sinking of the RMS Titanic is one of the most infamous shipwrecks in history.  On April 15, 1912, during her maiden voyage, the Titanic sank after colliding with an iceberg, killing 1502 out of 2224 passengers and crew. This sensational tragedy shocked the international community and led to better safety regulations for ships. One of the reasons that the shipwreck led to such loss of life was that there were not enough lifeboats for the passengers and crew. Although there was some element of luck involved in surviving the sinking, some groups of people were more likely to survive than others, such as women, children, and the upper-class. In this challenge, we ask you to complete the analysis of what sorts of people were likely to survive. In particular, we ask you to apply the tools of machine learning to predict which passengers survived the tragedy\". For more details, please refer to \"https://www.kaggle.com/c/titanic/overview\" \n",
    "\n",
    "The focus here is not the analysis of the data. Instead, we will concentrate on the identification of errors in the data, which might cause problems in the analysis. This data set contains the following variables\n",
    "* <font color=\"blue\">survival</font>: a boolean variable indicates whether the passenger survived or not.\n",
    "* <font color=\"blue\">pclass</font>: Passenger's Carbin Class \n",
    "* <font color=\"blue\">sex</font>: the gender of a passenger\n",
    "* <font color=\"blue\">age</font>: Age\n",
    "* <font color=\"blue\">sibsp</font>: Number of Siblings/Spouses Aboard \n",
    "* <font color=\"blue\">parch</font>: Number of Parents/Children Aboard \n",
    "* <font color=\"blue\">fare</font>: Passenger ticket Fare\n",
    "* <font color=\"blue\">embarked</font>: abbreviation of Port of Embarkation\n",
    "* <font color=\"blue\">class</font>: the passenger's carbin class\n",
    "* <font color=\"blue\">who</font>: a variable takes values in {man, woman, child}\n",
    "* <font color=\"blue\">adult_male</font>: a boolean variable\n",
    "* <font color=\"blue\">deck</font>: the deck\n",
    "* <font color=\"blue\">embark_town</font>: the name of the port of embarkation \n",
    "* <font color=\"blue\">alive</font>: whether or not the passenger was alive\n",
    "* <font color=\"blue\">alone</font>: a boolean variable indicates if the passenger traveled alone.\n",
    "* <font color=\"blue\">name</font>: Name of the passenger\n",
    "\n",
    "For convenience, we will use the demo version of Titanic data included in the seaborne library: https://github.com/mwaskom/seaborn-data/blob/master/titanic.csv. For the purpose of demonstration,\n",
    "some errors have been introduced in the data.\n",
    "Notice that this task was developed based on the materials provided in the Kaggle website. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "968af485",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T02:56:30.155178Z",
     "start_time": "2022-05-03T02:56:29.528538Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>body {\n",
       "    margin: 0;\n",
       "    font-family: Helvetica;\n",
       "}\n",
       "table.dataframe {\n",
       "    border-collapse: collapse;\n",
       "    border: none;\n",
       "}\n",
       "table.dataframe tr {\n",
       "    border: none;\n",
       "}\n",
       "table.dataframe td, table.dataframe th {\n",
       "    margin: 0;\n",
       "    border: 1px solid white;\n",
       "    padding-left: 0.25em;\n",
       "    padding-right: 0.25em;\n",
       "}\n",
       "table.dataframe th:not(:empty) {\n",
       "    background-color: #fec;\n",
       "    text-align: left;\n",
       "    font-weight: normal;\n",
       "}\n",
       "table.dataframe tr:nth-child(2) th:empty {\n",
       "    border-left: none;\n",
       "    border-right: 1px dashed #888;\n",
       "}\n",
       "table.dataframe td {\n",
       "    border: 2px solid #ccf;\n",
       "    background-color: #f4f4ff;\n",
       "}\n",
       "h3 {\n",
       "    color: white;\n",
       "    background-color: black;\n",
       "    padding: 0.5em;\n",
       "}\n",
       "</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Basic scientific python libs\n",
    "import pandas as pd\n",
    "#import numpy as np\n",
    "\n",
    "# Visualisation\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.pylab as pylab\n",
    "#import seaborn as sns\n",
    "\n",
    "# Configure visualisations\n",
    "%matplotlib inline\n",
    "mpl.style.use(\"ggplot\")\n",
    "\n",
    "#Notebook displace setting\n",
    "from IPython.core.display import HTML\n",
    "css = open('style/style-table.css').read() + open('style/style-notebook.css').read()\n",
    "\n",
    "HTML('<style>{}</style>'.format(css))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "742c039b",
   "metadata": {},
   "source": [
    "### First, load the data using Pandas library \n",
    "\n",
    "As we discussed in the lecture, the first thing you should do is to inpect the file and figure out the file format. It is not hard to see that the titanic data is store in a csv file. So we can use the <a href=\"http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html\">read_csv()</a> function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1364f6cf",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T02:57:01.191838Z",
     "start_time": "2022-05-03T02:57:01.175912Z"
    }
   },
   "outputs": [],
   "source": [
    "titanic = pd.read_csv(\"data/titanic_new.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68418571",
   "metadata": {},
   "source": [
    "Now, the data has been loaded and stored in a Panda DataFrame. We can take an overview of the data. For example, you might need to know \n",
    "* the number of columns, i.e., attributes, and what are the attributes?\n",
    "* the number of rows, i.e., passengers\n",
    "* what is the data type of each attributes?\n",
    "and etc.\n",
    "\n",
    "We start by looking at the dimensionality of the data and a few lines of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b51659d0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T02:57:24.092034Z",
     "start_time": "2022-05-03T02:57:24.068880Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(892, 16)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>survived</th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>embarked</th>\n",
       "      <th>class</th>\n",
       "      <th>who</th>\n",
       "      <th>adult_male</th>\n",
       "      <th>deck</th>\n",
       "      <th>embark_town</th>\n",
       "      <th>alive</th>\n",
       "      <th>alone</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C</td>\n",
       "      <td>First</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>C</td>\n",
       "      <td>Cherbourg</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>True</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>S</td>\n",
       "      <td>First</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>C</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.4583</td>\n",
       "      <td>Q</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Queenstown</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Moran, Mr. James</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>54.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>51.8625</td>\n",
       "      <td>S</td>\n",
       "      <td>First</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>E</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>McCarthy, Mr. Timothy J</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>21.0750</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>child</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Palsson, Master. Gosta Leonard</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>11.1333</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "      <td>Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>female</td>\n",
       "      <td>14.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>30.0708</td>\n",
       "      <td>C</td>\n",
       "      <td>Second</td>\n",
       "      <td>child</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Cherbourg</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "      <td>Nasser, Mrs. Nicholas (Adele Achem)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   survived  pclass     sex   age  sibsp  parch     fare embarked   class  \\\n",
       "0         0       3    male  22.0      1      0   7.2500        S   Third   \n",
       "1         1       1  female  38.0      1      0  71.2833        C   First   \n",
       "2         1       3  female  26.0      0      0   7.9250        S   Third   \n",
       "3         1       1  female  35.0      1      0  53.1000        S   First   \n",
       "4         0       3    male  35.0      0      0   8.0500        S   Third   \n",
       "5         0       3    male   NaN      0      0   8.4583        Q   Third   \n",
       "6         0       1    male  54.0      0      0  51.8625        S   First   \n",
       "7         0       3    male   2.0      3      1  21.0750        S   Third   \n",
       "8         1       3  female  27.0      0      2  11.1333        S   Third   \n",
       "9         1       2  female  14.0      1      0  30.0708        C  Second   \n",
       "\n",
       "     who  adult_male deck  embark_town alive  alone  \\\n",
       "0    man        True  NaN  Southampton    no  False   \n",
       "1  woman       False    C    Cherbourg   yes  False   \n",
       "2  woman       False  NaN  Southampton   yes   True   \n",
       "3  woman       False    C  Southampton   yes  False   \n",
       "4    man        True  NaN  Southampton    no   True   \n",
       "5    man        True  NaN   Queenstown    no   True   \n",
       "6    man        True    E  Southampton    no   True   \n",
       "7  child       False  NaN  Southampton    no  False   \n",
       "8  woman       False  NaN  Southampton   yes  False   \n",
       "9  child       False  NaN    Cherbourg   yes  False   \n",
       "\n",
       "                                                name  \n",
       "0                            Braund, Mr. Owen Harris  \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  \n",
       "2                             Heikkinen, Miss. Laina  \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  \n",
       "4                           Allen, Mr. William Henry  \n",
       "5                                   Moran, Mr. James  \n",
       "6                            McCarthy, Mr. Timothy J  \n",
       "7                     Palsson, Master. Gosta Leonard  \n",
       "8  Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)  \n",
       "9                Nasser, Mrs. Nicholas (Adele Achem)  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print (titanic.shape) \n",
    "titanic.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8c75133",
   "metadata": {},
   "source": [
    "You can also print out the last couple of rows with the <font color=\"blue\">tail()</font> function. We've got a sense of the variables, their class type, and the first few observations of each by observing the data. We know we're working with 892 observations of 15 variables. \n",
    "\n",
    "Next, we have a look at some key information about each variable to answer the following questions\n",
    "* **Which features are categorical?** These values classify the samples into sets of similar samples. Within categorical features are the values nominal, ordinal, ratio, or interval based? \n",
    "    * Categorical variables:<font color=\"blue\">survived</font>, <font color=\"blue\">sex</font>, <font color=\"blue\">embarked</font>, <font color=\"blue\">who</font>, <font color=\"blue\">embark_town</font>, <font color=\"blue\">alive</font>, <font color=\"blue\">alone</font>, and <font color=\"blue\">name</font>, <font color=\"blue\">deck</font>\n",
    "    * Ordinal variables: <font color=\"blue\">pclass</font>, <font color=\"blue\">class</font>, \n",
    "* **Which features are numerical**? Which features are numerical? Within numerical features are the values discrete, continuous, or time-series based?\n",
    "    * Continuous: <font color=\"blue\">age</font>, <font color=\"blue\">fare</font>. \n",
    "    * Discrete: <font color=\"blue\">sibsp</font>, <font color=\"blue\">parch</font>.\n",
    "\n",
    "Answering these questions will help us select the appropriate methods(e.g., plots) to audit the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "659641a1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T02:58:05.231835Z",
     "start_time": "2022-05-03T02:58:05.213774Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 892 entries, 0 to 891\n",
      "Data columns (total 16 columns):\n",
      " #   Column       Non-Null Count  Dtype  \n",
      "---  ------       --------------  -----  \n",
      " 0   survived     892 non-null    int64  \n",
      " 1   pclass       892 non-null    int64  \n",
      " 2   sex          892 non-null    object \n",
      " 3   age          715 non-null    float64\n",
      " 4   sibsp        892 non-null    int64  \n",
      " 5   parch        892 non-null    int64  \n",
      " 6   fare         892 non-null    float64\n",
      " 7   embarked     890 non-null    object \n",
      " 8   class        892 non-null    object \n",
      " 9   who          892 non-null    object \n",
      " 10  adult_male   892 non-null    bool   \n",
      " 11  deck         204 non-null    object \n",
      " 12  embark_town  890 non-null    object \n",
      " 13  alive        892 non-null    object \n",
      " 14  alone        892 non-null    bool   \n",
      " 15  name         892 non-null    object \n",
      "dtypes: bool(2), float64(2), int64(4), object(8)\n",
      "memory usage: 99.4+ KB\n"
     ]
    }
   ],
   "source": [
    "titanic.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd8c88bd",
   "metadata": {},
   "source": [
    "What is the distribution of the numerical values across the samples? This helps us determine, among other early insights, how representative is the training dataset of the actual problem domain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f4d4e69c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T02:58:51.011175Z",
     "start_time": "2022-05-03T02:58:50.952945Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>survived</th>\n",
       "      <th>pclass</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>892.000000</td>\n",
       "      <td>892.000000</td>\n",
       "      <td>715.000000</td>\n",
       "      <td>892.000000</td>\n",
       "      <td>892.000000</td>\n",
       "      <td>892.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.384529</td>\n",
       "      <td>2.307175</td>\n",
       "      <td>29.720517</td>\n",
       "      <td>0.522422</td>\n",
       "      <td>0.381166</td>\n",
       "      <td>32.201737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.486757</td>\n",
       "      <td>0.836750</td>\n",
       "      <td>14.490914</td>\n",
       "      <td>1.102264</td>\n",
       "      <td>0.805706</td>\n",
       "      <td>49.665589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.420000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>20.750000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7.917700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>28.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>14.454200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>31.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>512.329200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         survived      pclass         age       sibsp       parch        fare\n",
       "count  892.000000  892.000000  715.000000  892.000000  892.000000  892.000000\n",
       "mean     0.384529    2.307175   29.720517    0.522422    0.381166   32.201737\n",
       "std      0.486757    0.836750   14.490914    1.102264    0.805706   49.665589\n",
       "min      0.000000    1.000000    0.420000    0.000000    0.000000    0.000000\n",
       "25%      0.000000    2.000000   20.750000    0.000000    0.000000    7.917700\n",
       "50%      0.000000    3.000000   28.000000    0.000000    0.000000   14.454200\n",
       "75%      1.000000    3.000000   38.000000    1.000000    0.000000   31.000000\n",
       "max      1.000000    3.000000   80.000000    8.000000    6.000000  512.329200"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e024646",
   "metadata": {},
   "source": [
    "The observation tells us that \n",
    "* Total samples are 892 or 40% of the actual number of passengers on board the Titanic (2,224).\n",
    "* Survived is a categorical feature with 0 or 1 values.\n",
    "* Most passengers (> 75%) did not travel with parents or children.\n",
    "* Nearly 30% of the passengers had siblings and/or spouse aboard.\n",
    "* Fares varied significantly with few passengers paying as high as $512, and as low as 0.0.\n",
    "* Few elderly passengers within age range 65-80.\n",
    "* The minimum age is 0.42.\n",
    "* Some missing values exist in the \"age\" column\n",
    "\n",
    "Again, what is the distribution of the categorical variables?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ce259c22",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:00:23.295290Z",
     "start_time": "2022-05-03T03:00:23.262771Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sex</th>\n",
       "      <th>embarked</th>\n",
       "      <th>class</th>\n",
       "      <th>who</th>\n",
       "      <th>deck</th>\n",
       "      <th>embark_town</th>\n",
       "      <th>alive</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>892</td>\n",
       "      <td>890</td>\n",
       "      <td>892</td>\n",
       "      <td>892</td>\n",
       "      <td>204</td>\n",
       "      <td>890</td>\n",
       "      <td>892</td>\n",
       "      <td>892</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>891</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>male</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>C</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>Behr, Mr. Karl Howell</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>574</td>\n",
       "      <td>644</td>\n",
       "      <td>491</td>\n",
       "      <td>538</td>\n",
       "      <td>60</td>\n",
       "      <td>643</td>\n",
       "      <td>550</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         sex embarked  class  who deck  embark_town alive  \\\n",
       "count    892      890    892  892  204          890   892   \n",
       "unique     4        3      3    3    7            7     2   \n",
       "top     male        S  Third  man    C  Southampton    no   \n",
       "freq     574      644    491  538   60          643   550   \n",
       "\n",
       "                         name  \n",
       "count                     892  \n",
       "unique                    891  \n",
       "top     Behr, Mr. Karl Howell  \n",
       "freq                        2  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic.describe(include=['O'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ccb48df",
   "metadata": {},
   "source": [
    "What the table above tells us:\n",
    "* Names are not unique across the dataset (count !=unique), \"Behr, Mr. Karl Howell\" appears twice.\n",
    "* Sex variable as four possible values with 574 males, which is suspicious. \n",
    "* Embarked takes three possible values. In contrast, the embark_town takes 7 different values. However, if the two values represent the same information, it becomes suspicious.\n",
    "* Alive is a boolean variable\n",
    "* There are a lot of missing values in deck, and 2 in both <font color=\"blue\">embarked</font> and <font color=\"blue\">embark_town</font>.\n",
    "\n",
    "It is clear that the summary statistics on the distributions of each variable gives us a lot of information about the variable. Before we continue our auditing process, we are could further split the \"name\" column into more meaningful columns for better analysis. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d1c76e26",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:08:02.467374Z",
     "start_time": "2022-05-03T03:08:02.116134Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>survived</th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>embarked</th>\n",
       "      <th>class</th>\n",
       "      <th>who</th>\n",
       "      <th>adult_male</th>\n",
       "      <th>deck</th>\n",
       "      <th>embark_town</th>\n",
       "      <th>alive</th>\n",
       "      <th>alone</th>\n",
       "      <th>name</th>\n",
       "      <th>title</th>\n",
       "      <th>lastName</th>\n",
       "      <th>firstName</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Braund</td>\n",
       "      <td>Owen Harris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C</td>\n",
       "      <td>First</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>C</td>\n",
       "      <td>Cherbourg</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>Mrs</td>\n",
       "      <td>Cumings</td>\n",
       "      <td>John Bradley (Florence Briggs Thayer)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>True</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>Miss</td>\n",
       "      <td>Heikkinen</td>\n",
       "      <td>Laina</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>S</td>\n",
       "      <td>First</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>C</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>Mrs</td>\n",
       "      <td>Futrelle</td>\n",
       "      <td>Jacques Heath (Lily May Peel)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Allen</td>\n",
       "      <td>William Henry</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   survived  pclass     sex   age  sibsp  parch     fare embarked  class  \\\n",
       "0         0       3    male  22.0      1      0   7.2500        S  Third   \n",
       "1         1       1  female  38.0      1      0  71.2833        C  First   \n",
       "2         1       3  female  26.0      0      0   7.9250        S  Third   \n",
       "3         1       1  female  35.0      1      0  53.1000        S  First   \n",
       "4         0       3    male  35.0      0      0   8.0500        S  Third   \n",
       "\n",
       "     who  adult_male deck  embark_town alive  alone  \\\n",
       "0    man        True  NaN  Southampton    no  False   \n",
       "1  woman       False    C    Cherbourg   yes  False   \n",
       "2  woman       False  NaN  Southampton   yes   True   \n",
       "3  woman       False    C  Southampton   yes  False   \n",
       "4    man        True  NaN  Southampton    no   True   \n",
       "\n",
       "                                                name title   lastName  \\\n",
       "0                            Braund, Mr. Owen Harris    Mr     Braund   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...   Mrs    Cumings   \n",
       "2                             Heikkinen, Miss. Laina  Miss  Heikkinen   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)   Mrs   Futrelle   \n",
       "4                           Allen, Mr. William Henry    Mr      Allen   \n",
       "\n",
       "                               firstName  \n",
       "0                            Owen Harris  \n",
       "1  John Bradley (Florence Briggs Thayer)  \n",
       "2                                  Laina  \n",
       "3          Jacques Heath (Lily May Peel)  \n",
       "4                          William Henry  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Lets seperate the titles from the name \n",
    "coltitle = titanic['name'].apply(lambda s: pd.Series({'title': s.split(',')[1].split('.')[0].strip(),\n",
    "                                                      'lastName':s.split(',')[0].strip(), \n",
    "                                                      'firstName':s.split(',')[1].split('.')[1].strip()}))\n",
    "\n",
    "# Add the columns to the titanic dataframe\n",
    "titanic = pd.concat([titanic, coltitle], axis=1)\n",
    "\n",
    "#Drop the Name column, but here we choose to keep it at the moment.\n",
    "#titanic.drop('name', axis=1, inplace=True)\n",
    "titanic.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a937f9e",
   "metadata": {},
   "source": [
    "Notice that we kept the \"name\" column for now, as we might need to check where we have correctly split it into three columns."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb3f117c",
   "metadata": {},
   "source": [
    "### Identify Syntactical Anomalies \n",
    "In this section, we will demonstrate how to audit the data to identify some syntactical errors. \n",
    "\n",
    "#### Are all the titles consistent?\n",
    "\n",
    "Let's start with checking the <font color=\"blue\">title</font> column, as we just extract it from the <font color=\"blue\">name</font> column. We used the <font color=\"orange\">split()</font> function together with the following delimiters: \",\" and \".\" Is it possible that the split method we used gave us some erroneous extraction? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e236c446",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:09:18.163018Z",
     "start_time": "2022-05-03T03:09:18.154662Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Mr              518\n",
       "Miss            182\n",
       "Mrs             125\n",
       "Master           40\n",
       "Dr                7\n",
       "Rev               6\n",
       "Mlle              2\n",
       "Major             2\n",
       "Col               2\n",
       "the Countess      1\n",
       "Capt              1\n",
       "Ms                1\n",
       "Sir               1\n",
       "Lady              1\n",
       "Mme               1\n",
       "Don               1\n",
       "Jonkheer          1\n",
       "Name: title, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic.title.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "564c4049",
   "metadata": {},
   "source": [
    "We have got 17 different titles. We might ask if it is possible to have 17 different titles, as those often used are Mr, Mrs, Miss, Ms, and Dr. What is the meaning of the following title?\n",
    "* Rev \n",
    "* Mlle\n",
    "* Jonkheer\n",
    "* Don\n",
    "* Mme\n",
    "* The countess\n",
    "\n",
    "Is it possible that the patten we used to extract Title is not applicable to all the records?\n",
    "\n",
    "In this case, we might need to have a look at the rows whose title is equal to the tiles listed above. For example, we look at \"Rev\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "3e98dd97",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:10:14.485583Z",
     "start_time": "2022-05-03T03:10:14.458022Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>survived</th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>embarked</th>\n",
       "      <th>class</th>\n",
       "      <th>who</th>\n",
       "      <th>adult_male</th>\n",
       "      <th>deck</th>\n",
       "      <th>embark_town</th>\n",
       "      <th>alive</th>\n",
       "      <th>alone</th>\n",
       "      <th>name</th>\n",
       "      <th>title</th>\n",
       "      <th>lastName</th>\n",
       "      <th>firstName</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>42.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13.000</td>\n",
       "      <td>S</td>\n",
       "      <td>Second</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Byles, Rev. Thomas Roussel Davids</td>\n",
       "      <td>Rev</td>\n",
       "      <td>Byles</td>\n",
       "      <td>Thomas Roussel Davids</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>150</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>51.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12.525</td>\n",
       "      <td>S</td>\n",
       "      <td>Second</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Bateman, Rev. Robert James</td>\n",
       "      <td>Rev</td>\n",
       "      <td>Bateman</td>\n",
       "      <td>Robert James</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>249</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>54.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>26.000</td>\n",
       "      <td>S</td>\n",
       "      <td>Second</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Carter, Rev. Ernest Courtenay</td>\n",
       "      <td>Rev</td>\n",
       "      <td>Carter</td>\n",
       "      <td>Ernest Courtenay</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>626</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>57.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12.350</td>\n",
       "      <td>Q</td>\n",
       "      <td>Second</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Queenstown</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Kirkland, Rev. Charles Leonard</td>\n",
       "      <td>Rev</td>\n",
       "      <td>Kirkland</td>\n",
       "      <td>Charles Leonard</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>33.000</td>\n",
       "      <td>S</td>\n",
       "      <td>Second</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Harper, Rev. John</td>\n",
       "      <td>Rev</td>\n",
       "      <td>Harper</td>\n",
       "      <td>John</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>886</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13.000</td>\n",
       "      <td>S</td>\n",
       "      <td>Second</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Montvila, Rev. Juozas</td>\n",
       "      <td>Rev</td>\n",
       "      <td>Montvila</td>\n",
       "      <td>Juozas</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     survived  pclass   sex   age  sibsp  parch    fare embarked   class  who  \\\n",
       "149         0       2  male  42.0      0      0  13.000        S  Second  man   \n",
       "150         0       2  male  51.0      0      0  12.525        S  Second  man   \n",
       "249         0       2  male  54.0      1      0  26.000        S  Second  man   \n",
       "626         0       2  male  57.0      0      0  12.350        Q  Second  man   \n",
       "848         0       2  male  28.0      0      1  33.000        S  Second  man   \n",
       "886         0       2  male  27.0      0      0  13.000        S  Second  man   \n",
       "\n",
       "     adult_male deck  embark_town alive  alone  \\\n",
       "149        True  NaN  Southampton    no   True   \n",
       "150        True  NaN  Southampton    no   True   \n",
       "249        True  NaN  Southampton    no  False   \n",
       "626        True  NaN   Queenstown    no   True   \n",
       "848        True  NaN  Southampton    no  False   \n",
       "886        True  NaN  Southampton    no   True   \n",
       "\n",
       "                                  name title  lastName              firstName  \n",
       "149  Byles, Rev. Thomas Roussel Davids   Rev     Byles  Thomas Roussel Davids  \n",
       "150         Bateman, Rev. Robert James   Rev   Bateman           Robert James  \n",
       "249      Carter, Rev. Ernest Courtenay   Rev    Carter       Ernest Courtenay  \n",
       "626     Kirkland, Rev. Charles Leonard   Rev  Kirkland        Charles Leonard  \n",
       "848                  Harper, Rev. John   Rev    Harper                   John  \n",
       "886              Montvila, Rev. Juozas   Rev  Montvila                 Juozas  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic[titanic.title == \"Rev\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cf3ec5b",
   "metadata": {},
   "source": [
    "The are six rows in the DataFrame that contain \"Rev\". It seams that \"Rev\" is not a random lexical error, instead it might be a valid title that is not use often nowadays. We can check if \"Rev\" is a title by searching it online. What we will get from Wikipedia is\n",
    "> The Reverend is an honorific style most often placed before the names of Christian clergy and ministers. There are sometimes differences in the way the style is used in different countries and church traditions\n",
    "\n",
    "We can confirm that Rev is a title that is not often used nowadays. Similarly, you can check the other titles as well. It is interesting that \"The countess\" and \"Mlle\" are titles for female and \"Don\" for male. Should we unify these titles? For instance, assume that we are going to unify the title values by replacing \"The countess\", \"Lady\", \"Mme\" and \"Mlle\" with \"Miss\" and \"Don\" with \"Mr\", what should we do?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6a181695",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:11:38.117579Z",
     "start_time": "2022-05-03T03:11:38.111127Z"
    }
   },
   "outputs": [],
   "source": [
    "titanic.title.replace({'Mlle':'Miss', \n",
    "                       'the Countess':'Miss', \n",
    "                       'Lady':'Miss', \n",
    "                       'Mme':'Miss', \n",
    "                       'Don':'Mr'},inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9790729e",
   "metadata": {},
   "source": [
    "Now, we can drop the <font color=\"blue\">name</font> column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f5a09f37",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:12:04.946167Z",
     "start_time": "2022-05-03T03:12:04.939662Z"
    }
   },
   "outputs": [],
   "source": [
    "titanic.drop(['name'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0647f8f2",
   "metadata": {},
   "source": [
    "#### Are there any lexical errors in the data?\n",
    "Typos are the most common error, particularly whenever the data collection process involves human. While we were collecting the data, we might mis-type the name of the embark_town. It is always a good idea to check the categorical variables to make sure their values are spelled without errors. Let's look at the <font color=\"blue\">embark_town</font>. You can also use the <font color=\"blue\">value_counts()</font> function or the <font color=\"blue\">unique()</font> function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "edabd147",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:12:30.799525Z",
     "start_time": "2022-05-03T03:12:30.768537Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Southampton    643\n",
       "Cherbourg      167\n",
       "Queenstown      75\n",
       "queenstown       2\n",
       "Southamtpon      1\n",
       "Cherbourge       1\n",
       "Cherborg         1\n",
       "Name: embark_town, dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic.embark_town.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d473dafd",
   "metadata": {},
   "source": [
    "The output above show that \n",
    "* typos \n",
    "    * Cherbourg v.s. Cherborg\n",
    "    * Southampton v.s. Southamtpon\n",
    "    * Cherbourg v.s. Cherbourge\n",
    "* Inconsistent spelling:\n",
    " * Queenstown v.s. queenstown\n",
    "\n",
    "The assumption we made here is that it is less likely that the spelling with large counts is wrong. You can also check  <font color=\"blue\">embark_town</font> against <font color=\"blue\">embarked</font>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "909d19d0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:13:30.484173Z",
     "start_time": "2022-05-03T03:13:30.478325Z"
    }
   },
   "outputs": [],
   "source": [
    "titanic.embark_town.replace({\"Cherborg\": \"Cherbourg\", \n",
    "                             \"Cherbourge\": \"Cherbourg\", \n",
    "                             \"Southamtpon\":\"Southampton\", \n",
    "                             \"queenstown\":\"Queenstown\"}, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b727bac",
   "metadata": {},
   "source": [
    "The cross-tabulation of <font color=\"blue\">embark_town</font> and <font color=\"blue\">embarked</font> below proves the correspondence between the values of <font color=\"blue\">embark_town</font> and those of <font color=\"blue\">embarked</font>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a72b8b6a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:13:48.179591Z",
     "start_time": "2022-05-03T03:13:48.154394Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>embarked</th>\n",
       "      <th>C</th>\n",
       "      <th>Q</th>\n",
       "      <th>S</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>embark_town</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Cherbourg</th>\n",
       "      <td>168</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Queenstown</th>\n",
       "      <td>0</td>\n",
       "      <td>77</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Southampton</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>644</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "embarked       C   Q    S\n",
       "embark_town              \n",
       "Cherbourg    168   1    0\n",
       "Queenstown     0  77    0\n",
       "Southampton    0   0  644"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.crosstab(titanic[\"embark_town\"], titanic[\"embarked\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19084220",
   "metadata": {},
   "source": [
    "#### Further more, are their any other inconsistent spellings?\n",
    "Here, we are going to use the <font color=\"blue\">sex</font> variable as an example. The observation told us that it is a boolean variable that takes two values, i.e., <font color=\"blue\">male</font> and <font color=\"blue\">female</font> with lower case letters. Let's check is unique values in the <font color=\"blue\">sex</font> column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ba034ace",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:14:30.684339Z",
     "start_time": "2022-05-03T03:14:30.677305Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['male', 'female', 'M', 'F'], dtype=object)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic.sex.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00273074",
   "metadata": {},
   "source": [
    "The output shows the number of unique values in \"sex\" is 4, which is supposed to be 2. The inconsistency here is obvious. We can either replace male/female with M/F or M/F with male/female."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "f5d72fa7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:14:56.764659Z",
     "start_time": "2022-05-03T03:14:56.757775Z"
    }
   },
   "outputs": [],
   "source": [
    "titanic['sex'].replace({'F':'female', \n",
    "                        'M':'male'},inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f030703",
   "metadata": {},
   "source": [
    "You can check all the other categorical variables in a similar way."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7fc974f",
   "metadata": {},
   "source": [
    "### Semantic errors: \n",
    "\n",
    "Variables can be correlated with each other. One variable might provide information that we can use to validate another variable.  In this task, we will check where nor not the data\n",
    "* violate the integrity constraints\n",
    "* contradictions\n",
    "* duplication\n",
    "\n",
    "We firstly check the integrity constraints. Given the variable description, one can figure out that \"age\", \"who\" \n",
    "and \"adult_male\" are correlated. For example, **if we assume all the children should under 18, and both men and women should be 18 or above**,\n",
    "* Were all children's ages under 18? \n",
    "* Were the ages of all men and women greater than or equal to 18?\n",
    "\n",
    "To answer the questions, we need to compute the summary statistics individually for passengers marked as child, man and woman. One way is to use the <font color=\"blue\">describe()</font> function together with the <font color=\"blue\">groupby()</font> function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c9c1655d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:16:52.892601Z",
     "start_time": "2022-05-03T03:16:52.869884Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>who</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>child</th>\n",
       "      <td>83.0</td>\n",
       "      <td>6.598434</td>\n",
       "      <td>5.151912</td>\n",
       "      <td>0.42</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>man</th>\n",
       "      <td>414.0</td>\n",
       "      <td>33.155797</td>\n",
       "      <td>12.896093</td>\n",
       "      <td>16.00</td>\n",
       "      <td>23.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>80.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>woman</th>\n",
       "      <td>218.0</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>11.495441</td>\n",
       "      <td>16.00</td>\n",
       "      <td>23.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>63.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       count       mean        std    min   25%   50%   75%   max\n",
       "who                                                              \n",
       "child   83.0   6.598434   5.151912   0.42   2.0   5.0  10.0  25.0\n",
       "man    414.0  33.155797  12.896093  16.00  23.0  30.0  40.0  80.0\n",
       "woman  218.0  32.000000  11.495441  16.00  23.0  30.0  39.0  63.0"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic['age'].groupby(titanic['who']).describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05296764",
   "metadata": {},
   "source": [
    "The statistics shows that there are 83 children, 413 men and 218 women."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "09c694af",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:17:43.416156Z",
     "start_time": "2022-05-03T03:17:43.386305Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>survived</th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>embarked</th>\n",
       "      <th>class</th>\n",
       "      <th>who</th>\n",
       "      <th>adult_male</th>\n",
       "      <th>deck</th>\n",
       "      <th>embark_town</th>\n",
       "      <th>alive</th>\n",
       "      <th>alone</th>\n",
       "      <th>title</th>\n",
       "      <th>lastName</th>\n",
       "      <th>firstName</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>17.0</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "      <td>Miss</td>\n",
       "      <td>Andersson</td>\n",
       "      <td>Erna Alexandra</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>16.0</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>46.9000</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Miss</td>\n",
       "      <td>Goodwin</td>\n",
       "      <td>Lillian Amy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>female</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10.5000</td>\n",
       "      <td>S</td>\n",
       "      <td>Second</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>True</td>\n",
       "      <td>Miss</td>\n",
       "      <td>Ilett</td>\n",
       "      <td>Bertha</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>34.3750</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Ford</td>\n",
       "      <td>William Neal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>14.4583</td>\n",
       "      <td>C</td>\n",
       "      <td>Third</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Cherbourg</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Miss</td>\n",
       "      <td>Attalah</td>\n",
       "      <td>Malake</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>138</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9.2167</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Osen</td>\n",
       "      <td>Olaf Elon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>156</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.7333</td>\n",
       "      <td>Q</td>\n",
       "      <td>Third</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Queenstown</td>\n",
       "      <td>yes</td>\n",
       "      <td>True</td>\n",
       "      <td>Miss</td>\n",
       "      <td>Gilnagh</td>\n",
       "      <td>Katherine \"Katie\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>163</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Calic</td>\n",
       "      <td>Jovo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>208</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.7500</td>\n",
       "      <td>Q</td>\n",
       "      <td>Third</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Queenstown</td>\n",
       "      <td>yes</td>\n",
       "      <td>True</td>\n",
       "      <td>Miss</td>\n",
       "      <td>Carr</td>\n",
       "      <td>Helen \"Ellen\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>220</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Sunderland</td>\n",
       "      <td>Victor Francis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>266</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>39.6875</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Panula</td>\n",
       "      <td>Ernesti Arvid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>282</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9.5000</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>de Pelsmaeker</td>\n",
       "      <td>Alfons</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>307</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>17.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>108.9000</td>\n",
       "      <td>C</td>\n",
       "      <td>First</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>C</td>\n",
       "      <td>Cherbourg</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "      <td>Mrs</td>\n",
       "      <td>Penasco y Castellana</td>\n",
       "      <td>Victor de Satode (Maria Josefa Perez de Soto y...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>329</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>57.9792</td>\n",
       "      <td>C</td>\n",
       "      <td>First</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>B</td>\n",
       "      <td>Cherbourg</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "      <td>Miss</td>\n",
       "      <td>Hippach</td>\n",
       "      <td>Jean Gertrude</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>333</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>18.0000</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Vander Planke</td>\n",
       "      <td>Leo Edmondus</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>389</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>female</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12.0000</td>\n",
       "      <td>C</td>\n",
       "      <td>Second</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Cherbourg</td>\n",
       "      <td>yes</td>\n",
       "      <td>True</td>\n",
       "      <td>Miss</td>\n",
       "      <td>Lehmann</td>\n",
       "      <td>Bertha</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>433</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.1250</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Kallio</td>\n",
       "      <td>Nikolai Erland</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>500</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Calic</td>\n",
       "      <td>Petar</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>504</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>86.5000</td>\n",
       "      <td>S</td>\n",
       "      <td>First</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>B</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>True</td>\n",
       "      <td>Miss</td>\n",
       "      <td>Maioni</td>\n",
       "      <td>Roberta</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>532</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>17.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>7.2292</td>\n",
       "      <td>C</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Cherbourg</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Elias</td>\n",
       "      <td>Joseph Jr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>550</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>110.8833</td>\n",
       "      <td>C</td>\n",
       "      <td>First</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>C</td>\n",
       "      <td>Cherbourg</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Thayer</td>\n",
       "      <td>John Borland Jr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>574</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Rush</td>\n",
       "      <td>Alfred George John</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>721</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>17.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>7.0542</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Jensen</td>\n",
       "      <td>Svend Lauritz</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>746</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>20.2500</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Abbott</td>\n",
       "      <td>Rossmore Edward</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>764</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.7750</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Eklund</td>\n",
       "      <td>Hans Linus</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>781</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>17.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>57.0000</td>\n",
       "      <td>S</td>\n",
       "      <td>First</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>B</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "      <td>Mrs</td>\n",
       "      <td>Dick</td>\n",
       "      <td>Albert Adrian (Vera Gillespie)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>791</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>26.0000</td>\n",
       "      <td>S</td>\n",
       "      <td>Second</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Gaskell</td>\n",
       "      <td>Alfred</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>841</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10.5000</td>\n",
       "      <td>S</td>\n",
       "      <td>Second</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Mudd</td>\n",
       "      <td>Thomas Charles</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>844</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Culumovic</td>\n",
       "      <td>Jeso</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>853</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>39.4000</td>\n",
       "      <td>S</td>\n",
       "      <td>First</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>D</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "      <td>Miss</td>\n",
       "      <td>Lines</td>\n",
       "      <td>Mary Conover</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     survived  pclass     sex   age  sibsp  parch      fare embarked   class  \\\n",
       "68          1       3  female  17.0      4      2    7.9250        S   Third   \n",
       "71          0       3  female  16.0      5      2   46.9000        S   Third   \n",
       "84          1       2  female  17.0      0      0   10.5000        S  Second   \n",
       "86          0       3    male  16.0      1      3   34.3750        S   Third   \n",
       "114         0       3  female  17.0      0      0   14.4583        C   Third   \n",
       "138         0       3    male  16.0      0      0    9.2167        S   Third   \n",
       "156         1       3  female  16.0      0      0    7.7333        Q   Third   \n",
       "163         0       3    male  17.0      0      0    8.6625        S   Third   \n",
       "208         1       3  female  16.0      0      0    7.7500        Q   Third   \n",
       "220         1       3    male  16.0      0      0    8.0500        S   Third   \n",
       "266         0       3    male  16.0      4      1   39.6875        S   Third   \n",
       "282         0       3    male  16.0      0      0    9.5000        S   Third   \n",
       "307         1       1  female  17.0      1      0  108.9000        C   First   \n",
       "329         1       1  female  16.0      0      1   57.9792        C   First   \n",
       "333         0       3    male  16.0      2      0   18.0000        S   Third   \n",
       "389         1       2  female  17.0      0      0   12.0000        C  Second   \n",
       "433         0       3    male  17.0      0      0    7.1250        S   Third   \n",
       "500         0       3    male  17.0      0      0    8.6625        S   Third   \n",
       "504         1       1  female  16.0      0      0   86.5000        S   First   \n",
       "532         0       3    male  17.0      1      1    7.2292        C   Third   \n",
       "550         1       1    male  17.0      0      2  110.8833        C   First   \n",
       "574         0       3    male  16.0      0      0    8.0500        S   Third   \n",
       "721         0       3    male  17.0      1      0    7.0542        S   Third   \n",
       "746         0       3    male  16.0      1      1   20.2500        S   Third   \n",
       "764         0       3    male  16.0      0      0    7.7750        S   Third   \n",
       "781         1       1  female  17.0      1      0   57.0000        S   First   \n",
       "791         0       2    male  16.0      0      0   26.0000        S  Second   \n",
       "841         0       2    male  16.0      0      0   10.5000        S  Second   \n",
       "844         0       3    male  17.0      0      0    8.6625        S   Third   \n",
       "853         1       1  female  16.0      0      1   39.4000        S   First   \n",
       "\n",
       "       who  adult_male deck  embark_town alive  alone title  \\\n",
       "68   woman       False  NaN  Southampton   yes  False  Miss   \n",
       "71   woman       False  NaN  Southampton    no  False  Miss   \n",
       "84   woman       False  NaN  Southampton   yes   True  Miss   \n",
       "86     man        True  NaN  Southampton    no  False    Mr   \n",
       "114  woman       False  NaN    Cherbourg    no   True  Miss   \n",
       "138    man        True  NaN  Southampton    no   True    Mr   \n",
       "156  woman       False  NaN   Queenstown   yes   True  Miss   \n",
       "163    man        True  NaN  Southampton    no   True    Mr   \n",
       "208  woman       False  NaN   Queenstown   yes   True  Miss   \n",
       "220    man        True  NaN  Southampton   yes   True    Mr   \n",
       "266    man        True  NaN  Southampton    no  False    Mr   \n",
       "282    man        True  NaN  Southampton    no   True    Mr   \n",
       "307  woman       False    C    Cherbourg   yes  False   Mrs   \n",
       "329  woman       False    B    Cherbourg   yes  False  Miss   \n",
       "333    man        True  NaN  Southampton    no  False    Mr   \n",
       "389  woman       False  NaN    Cherbourg   yes   True  Miss   \n",
       "433    man        True  NaN  Southampton    no   True    Mr   \n",
       "500    man        True  NaN  Southampton    no   True    Mr   \n",
       "504  woman       False    B  Southampton   yes   True  Miss   \n",
       "532    man        True  NaN    Cherbourg    no  False    Mr   \n",
       "550    man        True    C    Cherbourg   yes  False    Mr   \n",
       "574    man        True  NaN  Southampton    no   True    Mr   \n",
       "721    man        True  NaN  Southampton    no  False    Mr   \n",
       "746    man        True  NaN  Southampton    no  False    Mr   \n",
       "764    man        True  NaN  Southampton    no   True    Mr   \n",
       "781  woman       False    B  Southampton   yes  False   Mrs   \n",
       "791    man        True  NaN  Southampton    no   True    Mr   \n",
       "841    man        True  NaN  Southampton    no   True    Mr   \n",
       "844    man        True  NaN  Southampton    no   True    Mr   \n",
       "853  woman       False    D  Southampton   yes  False  Miss   \n",
       "\n",
       "                 lastName                                          firstName  \n",
       "68              Andersson                                     Erna Alexandra  \n",
       "71                Goodwin                                        Lillian Amy  \n",
       "84                  Ilett                                             Bertha  \n",
       "86                   Ford                                       William Neal  \n",
       "114               Attalah                                             Malake  \n",
       "138                  Osen                                          Olaf Elon  \n",
       "156               Gilnagh                                  Katherine \"Katie\"  \n",
       "163                 Calic                                               Jovo  \n",
       "208                  Carr                                      Helen \"Ellen\"  \n",
       "220            Sunderland                                     Victor Francis  \n",
       "266                Panula                                      Ernesti Arvid  \n",
       "282         de Pelsmaeker                                             Alfons  \n",
       "307  Penasco y Castellana  Victor de Satode (Maria Josefa Perez de Soto y...  \n",
       "329               Hippach                                      Jean Gertrude  \n",
       "333         Vander Planke                                       Leo Edmondus  \n",
       "389               Lehmann                                             Bertha  \n",
       "433                Kallio                                     Nikolai Erland  \n",
       "500                 Calic                                              Petar  \n",
       "504                Maioni                                            Roberta  \n",
       "532                 Elias                                          Joseph Jr  \n",
       "550                Thayer                                    John Borland Jr  \n",
       "574                  Rush                                 Alfred George John  \n",
       "721                Jensen                                      Svend Lauritz  \n",
       "746                Abbott                                    Rossmore Edward  \n",
       "764                Eklund                                         Hans Linus  \n",
       "781                  Dick                     Albert Adrian (Vera Gillespie)  \n",
       "791               Gaskell                                             Alfred  \n",
       "841                  Mudd                                     Thomas Charles  \n",
       "844             Culumovic                                               Jeso  \n",
       "853                 Lines                                       Mary Conover  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic[((titanic.who == \"man\") | (titanic.who == \"woman\")) & (titanic.age < 18)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fc842c7",
   "metadata": {},
   "source": [
    "There are 30 passenger should be classified as man and woman respectively. Now we can replace the value of <font color=\"blue\">who</font> for the above record to \"child\", given the assumption we made."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "7c9b196d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:18:23.369078Z",
     "start_time": "2022-05-03T03:18:23.362338Z"
    }
   },
   "outputs": [],
   "source": [
    "titanic.loc[((titanic.who == \"man\") | (titanic.who == \"woman\")) & (titanic.age < 18), \"who\"] = \"child\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c300b2fb",
   "metadata": {},
   "source": [
    "Instead of using the <font color=\"orange\">describe()</font> function, you can also choose to use plot. For example,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "acea33ff",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:18:45.136513Z",
     "start_time": "2022-05-03T03:18:44.736006Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[<AxesSubplot:title={'center':'child'}>,\n",
       "        <AxesSubplot:title={'center':'man'}>],\n",
       "       [<AxesSubplot:title={'center':'woman'}>, <AxesSubplot:>]],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAELCAYAAAA4HCbKAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAgWklEQVR4nO3dfVRUdf4H8PcwKM8PA8NDjk+LZASCboLPHQSm2mO6TZ7WUx7cKPtttrtqopxQfKCMxBQpPKbtnjK33f3DOjrtlpvHUSAfzuaYcSRME/Oh0uRpACcEhLm/PzzNSqAwl3n+vl9/ce/M3O/nXpg3996587kKSZIkEBGRV/NxdQFEROR4DHsiIgEw7ImIBMCwJyISAMOeiEgADHsiIgEw7F1k9OjRePXVV+/6nJycHGi1Wut0YWEh4uPj7/qaiooKKBQKfP/993apk4i8g6+rC6A7e/PNN2GxWFxdBhF5AYa9GwsLC3N1CUTkJXgax8G2bduGxMRE+Pn5ITo6Gk888YT1sc7OTixduhQRERGIiYnBihUr0N3dbX38l6dx+rJ161YMHz4cgYGBeOSRR3D58mWHrQvRYMycORMLFy7E6tWrER0djfDwcBQUFMBiseCVV15BTEwMoqKiUFBQYH3NP//5T0yePBlhYWFQq9V49NFH8c0331gfv3jxIhQKBXbv3o05c+YgMDAQcXFxeP/9912xim6NYe9A69atw0svvYQ//vGPqK6uxqeffooJEyZYH9+6dSvuuecefP755ygrK8Mbb7yBv/3tbwNe/kcffYRly5YhNzcXVVVVmDdvHvLy8hywJkT28eGHH+LmzZs4cuQItmzZgtdeew2zZ8+G2WzG4cOHsXnzZrz22mv4z3/+AwDo6OjAmjVrcPLkSRw4cABKpRKPPvooOjs7eyw3Pz8fCxYswKlTpzBv3jw888wzOHfunCtW0X1J5BBms1ny9/eXNm3a1Ofjo0aNkubMmdNj3iOPPCI9+eST1umnn35aysrKsk6vW7dOGjNmjHV6+vTp0vz583ssY/ny5RIA6bvvvrPHahDZTXp6ujR+/Pge8xITE6Vx48b1mJeSkiItX768z2U0NjZKAKQjR45IkiRJFy5ckABIJSUl1ufcvHlTCgoKknbs2GHfFfBw3LN3kJqaGrS3t+Phhx++43Nu38sHAI1Gg2vXrg14jNOnT2PatGk95s2YMcOmOomcafz48T2mY2NjkZKS0mteXV0dAKCqqgqPP/44fvWrXyEkJAQjR44EAFy6dKnHa25/L/n6+iImJsam95IIGPYOplAo7vjY0KFDez3X1qtv7rZ8InczZMiQHtMKhaLPeRaLBW1tbXj44YehUCjw7rvv4vjx4zAajVAoFL1O49jjveTtGPYOkpiYCH9/f+zfv9+hYxw9erTHvF9OE3mqr7/+GvX19SgqKkJGRgbuv/9+mEwmSOzKLgsvvXSQ4OBgLF++HIWFhQgICMBDDz2EGzduYN++fVi5cqVdxli+fDl+97vfYdKkSZg1axaOHDnCqxDIa4waNQp+fn7YunUrli9fjosXLyI/P59HszJxz96B1q9fj6KiIpSVlWHcuHF4+OGHcfLkSbst//HHH0dJSQlef/11pKSk4B//+Ac2btxot+UTuZJarcbf//53HDhwAElJSVixYgU2b94MHx/GlhwKicdERERej/8iiYgEwLAnIhIAw56ISAAMeyIiATDsiYgE0O919p2dnVi3bh26urrQ3d2NKVOmYN68eTCbzSgtLUV9fT2ioqKwbNkyBAcHO6NmIiKyUb+XXkqShI6ODvj7+6Orqwtr165FTk4Ojh8/juDgYOh0Ouj1epjNZmRnZ/c74JUrV6w/q9VqNDQ0DH4t3BTXzzmGDRvm6hJkuf290B932dYDxXod62713un90O9pHIVCAX9/fwBAd3c3uru7oVAoYDQakZ6eDgBIT0+H0WiUWzeRW3nrrbfw3HPPYfny5dZ5ZrMZ69evx5IlS7B+/XqYzWbrY3v37sXixYuxdOlSVFVVuaBiov4N6Jy9xWJBXl4ennvuOSQnJ+Pee+9FS0sLVCoVAEClUqG1tdWhhRI5y8yZM7Fq1aoe8/R6PZKTk1FWVobk5GTo9XoAwPfff49jx45hy5YtKCgowDvvvMMGXOSWBtQbx8fHB5s2bcJPP/2EzZs323Q3JIPBAIPBAAAoLi6GWq3+3+C+vj2mvY0r1u/a49P6f9IvxOw9Jmssb/39JSYmWlvs/sxoNKKwsBDArSPZwsJCZGdnw2g0Ytq0aRgyZAiio6MRGxuL2tpajB07Vvb43f/3217zBtKsV/nXf8kek7yfTY3QgoKCkJiYiKqqKoSFhcFkMkGlUsFkMiE0NLTP12i12h631rv9PJOnnSezlaesn9wa3WX9nHHO/k5Hsk1NTbj33nutz4uIiEBTU5PD6yGyVb9h39raCqVSiaCgIHR2dqK6uhqPPfYYUlNTUVlZCZ1Oh8rKSqSlpTmjXiK3Yktrqbsd5d5O7i033Okoy9OO+kSot9+wN5lM2LZtGywWCyRJwtSpUzFx4kSMHTsWpaWlOHToENRqNXJzc2UXTuTu7nQkGxkZicbGRuvzmpqaEBER0ecy7naUaw/ucJT1M3c56hsob6r3Tke6/Yb9qFGj8Prrr/eaHxISgrVr19pYIpFnutORbGpqKsrKyjB79myYTCZcvXoV8fHxLq6WqDfevIToF9544w2cPn0a169fx6JFizBv3jzodLo+j2RHjBiBqVOnIjc3Fz4+Pli4cCH7rZNbYtgT/cKLL77Y5/w7HcnOnTsXc+fOdWBFRIPHXRAiIgFwz57IS/R1fX5/eG2+OLhnT0QkAIY9EZEAGPZERAJg2BMRCYBhT0QkAIY9EZEAGPZERAJg2BMRCcBtvlTFL4QQETkO9+yJiATAsCciEgDDnohIAAx7IiIBMOyJiATgNlfjEJHzybkKDuCVcJ6o37BvaGjAtm3b0NzcDIVCAa1Wi1mzZsFsNqO0tBT19fWIiorCsmXLEBwc7IyaiYjIRv2GvVKpxIIFCxAXF4cbN24gPz8fKSkpqKioQHJyMnQ6HfR6PfR6PbKzs51RMxER2ajfc/YqlQpxcXEAgICAAGg0GjQ1NcFoNCI9PR0AkJ6eDqPR6NhKiYhINps+oK2rq8OFCxcQHx+PlpYWqFQqALf+IbS2tjqkQCIiGrwBf0Db3t6OkpIS5OTkIDAwcMADGAwGGAwGAEBxcTHUavX/Bvf1tU5fG/AS/0fuh0sxe4/Jep2tbl8/Z5GzHeXW6Ir1IyJ5BhT2XV1dKCkpwYMPPojJkycDAMLCwmAymaBSqWAymRAaGtrna7VaLbRarXW6oaHB+rNare4x7SzOGtNV62cruTW6y/oNGzbM1SUQub1+T+NIkoQdO3ZAo9Fg9uzZ1vmpqamorKwEAFRWViItLc1xVRIR0aD0u2d/9uxZfPbZZxg5ciTy8vIAAE899RR0Oh1KS0tx6NAhqNVq5ObmOrxYIiKSp9+wT0hIwO7du/t8bO3atXYviIiI7I/tEoiIBMCwJyISAMOeiEgADHsiIgEw7ImIBMAWx0Q2+NOf/gR/f3/4+PhAqVSiuLiYHWDJIzDsiWy0bt26Ht8Y1+v17ABLbo+ncYgGiR1gyRNwz57IRkVFRQCAhx56CFqtlh1gySMw7IlssH79ekRERKClpQWvvvqqTU3Y7tYB9nZyOpc6W3/dTj2tI6oI9TLsiWwQEREB4FbX17S0NNTW1tqlA6yn6a92d+mIOlDeVO+ddkAY9jawtX/+Nci/MbPcXv3kOO3t7ZAkCQEBAWhvb8epU6fwxBNPWDvA6nQ6doAlt8WwJxqglpYWbN68GQDQ3d2NGTNmYMKECRgzZgw7wJLbY9gTDVBMTAw2bdrUa35ISAg7wJLb46WXREQCYNgTEQmAYU9EJACesycim/V3tdidvisg9+o0Grx+w/6tt97CyZMnERYWhpKSEgBg4yciIg/T72mcmTNnYtWqVT3m/dz4qaysDMnJydDr9Y6qj4iI7KDfsE9MTOy1187GT0REnkXWB7Rs/ERE5Fkc/gHt3Zo/3d7Mx5nNn+Q2PJJTozPHkktujZ7WPIpIZLLCfqCNn4C7N39yVfMhZ47pCc2V5NboLs2jbOk8SSQqWWHPxk9EJIecBn+8XNM++g37N954A6dPn8b169exaNEizJs3Dzqdjo2fiIg8SL9h/+KLL/Y5n42fiIg8B9slEBEJgGFPRCQA9sYhIrcm965t/GC3J+7ZExEJgGFPRCQAhj0RkQAY9kREAmDYExEJgGFPRCQAIS+9lHspl7uPRUR0J0KGPRF5P1t2tG5vKe6t1+fzNA4RkQAY9kREAmDYExEJgGFPRCQAfkBLRHQbb72bFvfsiYgEwLAnIhIAT+MQEQ2SJ/TcH1TYV1VVYefOnbBYLMjKyoJOp7NTWUSeh+8Hcmeyw95iseCdd97B6tWrERkZiZUrVyI1NRXDhw+3Z33khbzxAzC+H8jdyQ772tpaxMbGIiYmBgAwbdo0GI1G/nGTkPh+IDlk987ae8zml8gO+6amJkRGRlqnIyMjce7cuV7PMxgMMBgMAIDi4mIMGzasx+PW6U9OyC2FXOiXv88B8cLf9UDeD/29F6y8cPuQ/dn63pN9NY4kSb3mKRSKXvO0Wi2Ki4tRXFzc67H8/Hy5w3sErp84BvJ+uNt7oT+etq1Zr2PJqVd22EdGRqKxsdE63djYCJVKJXdxRB6N7wdyd7LDfsyYMbh69Srq6urQ1dWFY8eOITU11Z61EXkMvh/I3ck+Z69UKvHss8+iqKgIFosFGRkZGDFihE3L0Gq1cof3CFw/cdjj/XA3nratWa9jyalXIfV1spGIiLwK2yUQEQmAYU9EJACGPRGRAJzaCO2HH36A0WhEU1MTFAoFVCoVv1JOROQETvuAVq/X4+jRo5g+fToiIiIA3PrW4c/z2DTKfbW1tWHv3r0wGo1obW0FAISFhSE1NRU6nQ5BQUEurtB7eNq2Zr2OZc96nRb2S5cuRUlJCXx9ex5MdHV1ITc3F2VlZc4ow2E87Y/IFkVFRUhKSsLMmTMRHh4OAGhubkZFRQWqq6uxZs0a1xboRTxtW7Nex7JnvU47Z69QKGAymXrNN5lMfbZZ8DSlpaUICgpCYWEh3n33Xbz77rtYt24dgoKCsGXLFleXNyh1dXXQ6XTWPzYACA8Ph06nQ0NDg+sK80Ketq1Zr2PZs16nnbPPycnBK6+8gnvuucfaMKqhoQE//vgjFi5c6KwyHKaurg4FBQU95v38SykvL3dRVfYRFRWFjz76COnp6b32LtRqtWuL8zKetq1Zr2PZs16nfqnKYrGgtrYWTU1NAICIiAjEx8fDx8fzLwp69dVXkZyc3OcvxR0PD21hNpuh1+tx4sQJtLS0ALj1j2zixInQ6XQIDg52cYXew9O2Net1LHvWy2/Q2omn/RHZ6ocffkBjYyPGjh0Lf39/6/yqqipMmDDBdYV5IU/f1lu3bsXixYtdXUafzp07B41Gg8DAQHR0dECv1+PChQsYPnw45s6di8DAQFeX2ENXVxeOHDmCiIgIpKSk4PDhwzh79iyGDx8OrVbb6zPQu2HYO0F5eTkyMjJcXYZs+/btw/79+6HRaHDp0iXk5OQgLS0NAPDSSy9h48aNLq7Qe3jatu6rnq+++grjxo0DcKtmd5Kbm4tNmzZBqVTi7bffhp+fH6ZMmYLq6mpcunQJK1ascHWJPZSVlaG7uxudnZ3Wf1CTJk1CdXU1JEnCn//85wEvizccd4Ldu3d7dNgfPHgQGzduhL+/P+rq6rBlyxbU19dj1qxZffZxJ/k8bVs3NTVBo9EgKysLCoUCkiTh/PnzmDNnjqtL65MkSVAqlQCAb7/91vrPKiEhAXl5ea4srU+XL1/G5s2b0d3djUWLFuHtt9+Gj48PHnzwQZvrZdjbyZ32CCRJsp7W8VQWi8V6OiE6OhqFhYUoKSlBfX29WwaQJ/O0bb1hwwbs27cPe/bswYIFCzB69GgMHToUiYmJri6tTyNGjLAeaY8aNQrnz5/HmDFjcOXKFZtOiTiLJEno6upCe3s7Ojo60NbWhuDgYNy8eRPd3d02Lcv91s5DtbS0oKCgoNf19JIkefSHs8Ctzx4uXryI0aNHAwD8/f2Rn5+P7du34/Lly64tzst42rb28fHB7NmzMXXqVOzatQthYWE2h5AzLVq0CDt37sSePXsQEhJivUF8ZGQknn/+eVeX10tGRgZefPFFWCwWPPnkk9iyZQuio6Nx7tw5TJs2zaZl8Zy9nWzfvh0ZGRlISEjo9dibb76JpUuXuqAq+2hsbIRSqexxre/Pzpw50+c6kzyevq1PnjyJM2fOYP78+a4u5a5u3LiBa9euwWKxICIios/t7S5uv3rxp59+QnV1NdRqNeLj421aDsOeiEgAnn+BOxER9YthT0QkAIY9EZEAGPZERAJg2BMRCYBhT0QkAIY9EZEAGPZERAJg2MtgMBjg5+eHtrY2AEB7ezv8/f0xY8YM63PKy8vh6+uL1tZWXL16FU8++STCw8MREBCAmTNn4sSJE9bnVlRUQKFQYN++fZg6dSoCAgIwceJE1NTUoKamBjNmzEBgYCAmTZqE06dPW19nMpmQnZ2NkSNHIiAgAPfddx9KSkp69FDJycmBVqvFX/7yF4waNQqhoaF47LHHUF9f74QtRUTugmEvw/Tp06FQKHD48GEAwNGjRxESEoLjx4/DbDYDAA4dOoTU1FSEhIRAp9PhzJkz+Pjjj3H8+HHExMTgoYce6nVbsYKCAhQVFeGLL77A0KFD8dRTT+GFF17Ayy+/bJ33zDPPWJ/f0dGB5ORk6PV6nD59GmvWrMG6devw3nvv9Viu0WhEeXk5PvnkE3z66aeoqqpyu1auRORgEsmSnp4u5eXlSZIkSatWrZKeffZZ6f7775c++eQTSZIkadq0adLKlSslg8EgAZBqamqsr21vb5diY2Oll19+WZIkSSovL5cASHv37rU+Z/fu3RIA6cMPP7TO27NnjwRAun79+h3rWrJkiaTVaq3TTz/9tKRWq6X29nbrvA0bNkixsbGD2wBE5FG4Zy9TZmYmDh06BODWXnxWVhYyMjJw6NAhmM1mGI1GZGZmoqamBpGRkT1avvr5+WHy5Mmoqanpsczx48dbf46NjQUApKSk9JpXV1cH4FY73OLiYkyYMAFqtRrBwcHYsWMHLl261GO5999/P/z8/KzTGo0G165ds8dmICIPwbCXKTMzE19++SUuX76ML774ApmZmcjMzMTBgwdx+PBh+Pj4YPr06QAAhULR6/WSJPWaP2TIEOvPPz/W1zyLxQIAKCkpwYYNG7B48WIcOHAAVVVVeO6559DZ2dljuUOHDu0x/fNNJohIHAx7mSZPnoyAgAC88soruPfeexEbG4uMjAxUV1fjgw8+wJQpUxAQEICkpCQ0NDT0+GC1o6MDx48fR1JS0qBq+Oyzz/Cb3/wGCxcuxK9//WvEx8fj3Llzg101IvJCDHuZhgwZghkzZmDXrl3IzMwEcKvfdHJyMt5//33rvMzMTEyaNAnz58/H0aNH8dVXX+H3v/892tvb8cILLwyqhvvuuw8VFRUoLy/HN998g9WrV+Pzzz8f9LoRkfdh2A9CVlYWurq6rMEO3Ar32+cpFAro9XokJCTg0UcfRVpaGn788UccOHAAarV6UOOvWbMG6enpeOyxxzB16lSYTCYsWbJkUMskIu/Em5cQEQmAe/ZERAJg2BMRCYBhT0QkAIY9EZEAGPZERAJg2BMRCcDX2QNeuXLFpuer1epe3SGdwVXjunJsTx132LBhdqyGyDsNOOwtFgvy8/MRERGB/Px8mM1mlJaWor6+HlFRUVi2bBmCg4MdWSsREck04NM4+/btg0ajsU7r9XokJyejrKzM2lOdiIjc04D27BsbG3Hy5EnMnTsXH3/8MYBbN8QoLCwEAKSnp6OwsBDZ2dmyC+n+v9/2Of9ujXiVf/2X7PGIiEQyoD379957D9nZ2T1a8ra0tEClUgEAVCoVWltbHVMhERENWr979l988QXCwsIQFxfX62YbA2EwGGAwGAAAxcXFd2z+JedWGoNtJHY3vr6+Dl2+O44t2rhEIuk37M+ePYsTJ07gyy+/RGdnJ27cuIGysjKEhYXBZDJBpVLBZDIhNDS0z9drtVpotVrrtD2v9nDklSO8GsdzxuXVOET96zfs58+fj/nz5wMAampq8O9//xtLlizB+++/j8rKSuh0OlRWViItLc3hxdrLnT4fuF1fRxr8jICIPJXsL1XpdDqcOnUKS5YswalTp6DT6exYFhER2ZNNX6pKSkqy3kovJCQEa9eudUhRRERkX2yXQEQkAIY9EZEAGPZERAJg2BMRCYBhT0QkAIY9EZEAGPZERAJg2BMRCYBhT0QkAIY9EZEAGPZERAJg2BMRCYBhT0QkAJu6XrqbgfSlJyIi7tkTEQmBYU9EJACGPRGRABj2REQCYNgTEQmAYU9EJACGPRGRABj2REQCYNgTEQmAYU9EJACGPRGRADy6N44nkNO/5xoA5V//Zf9iiEhY3LMnIhIAw56ISAAMeyIiATDsiYgEwLAnIhJAv1fjNDQ0YNu2bWhuboZCoYBWq8WsWbNgNptRWlqK+vp6REVFYdmyZQgODnZGzS7DO2MRkafqN+yVSiUWLFiAuLg43LhxA/n5+UhJSUFFRQWSk5Oh0+mg1+uh1+uRnZ3tjJqJiMhG/Z7GUalUiIuLAwAEBARAo9GgqakJRqMR6enpAID09HQYjUbHVkpERLLZdM6+rq4OFy5cQHx8PFpaWqBSqQDc+ofQ2trqkAKJiGjwBvwN2vb2dpSUlCAnJweBgYEDHsBgMMBgMAAAiouLoVar+3zetQEvUQx32k6O5OvrK9S4RCIZUNh3dXWhpKQEDz74ICZPngwACAsLg8lkgkqlgslkQmhoaJ+v1Wq10Gq11umGhgY7lO39XLGd1Gq1R447bNgwO1ZD5J36PY0jSRJ27NgBjUaD2bNnW+enpqaisrISAFBZWYm0tDTHVUlERIPS75792bNn8dlnn2HkyJHIy8sDADz11FPQ6XQoLS3FoUOHoFarkZub6/BiiYhInn7DPiEhAbt37+7zsbVr19q9ICIisj9+g5aISAAMeyIiATDsiYgEwLAnIhIAw56ISAAMeyIiATDsiYgEMODeOOQZ5PTcV/71Xw6ohIjcCffsiYgEwLAnIhIAw56ISAA8Z++meL9bIrIn7tkTEQmAYU9EJACGPRGRABj2REQCYNgTEQmAV+OQ9cqfa04aj9/YJXI+7tkTEQmAYU9EJACGPRGRABj2REQCYNgTEQmAYU9EJACGPRGRABj2REQCYNgTEQmA36Alp/tlr/6BfHOX37olGhzu2RMRCYBhT0QkAIY9EZEABnXOvqqqCjt37oTFYkFWVhZ0Op2dyiIiInuSvWdvsVjwzjvvYNWqVSgtLcXRo0fx/fff27M2IiKyE9lhX1tbi9jYWMTExMDX1xfTpk2D0Wi0Z21ERGQnssO+qakJkZGR1unIyEg0NTXZpSgiIrIv2efsJUnqNU+hUPSaZzAYYDAYAADFxcUYNmxY3wv85ITcUoiIqB+y9+wjIyPR2NhonW5sbIRKper1PK1Wi+LiYhQXF8saJz8/X26Jg+KqcV05tmjjEolEdtiPGTMGV69eRV1dHbq6unDs2DGkpqbaszYiIrIT2adxlEolnn32WRQVFcFisSAjIwMjRoywZ21ERGQng7rO/oEHHsADDzxgr1r6pNVqHbp8dxvXlWOLNi6RSBRSX5+0EhGRV2G7BCIiATDsiYgEwLAnIhIAw56ISABudaeqtrY27N27F0ajEa2trQCAsLAwpKamQqfTISgoiON6+NiuXGcikbnV1ThFRUVISkrCzJkzER4eDgBobm5GRUUFqqursWbNGo7r4WO7cp2JROZWp3Hq6uqg0+msIQAA4eHh0Ol0aGho4LheMLYr15lIZG4V9lFRUfjoo4/Q3Nxsndfc3Ay9Xg+1Ws1xvWBsV64zkcjc6jSO2WyGXq/HiRMn0NLSAuDWXt/EiROh0+kQHBzMcT18bFeuM5HI3CrsAeCHH35AY2Mjxo4dC39/f+v8qqoqTJgwwWHj1tbWAgDi4+Px3XffoaqqChqNxuHtIH5p69atWLx4sVPHBICvv/4atbW1GDlyJMaPH++wcc6dOweNRoPAwEB0dHRAr9fjwoULGD58OObOnYvAwECHjU0kMrcK+3379mH//v3QaDS4dOkScnJykJaWBgB46aWXsHHjRoeM+8EHH6Cqqgrd3d1ISUlBbW0tEhMTUV1djfHjx2Pu3LkOGbev9fnqq68wbtw4ALfW2VFWrlyJDRs2AAAOHjyI/fv3Iy0tDadOnbLuZTtCbm4uNm3aBKVSibfffht+fn6YMmUKqqurcenSJaxYscIh4xKJzq0uvTx48CA2btwIf39/1NXVYcuWLaivr8esWbP6vFmKvfz3v//Fpk2bcPPmTfzhD3/A9u3bERgYiN/+9rdYtWqVw8K+qakJGo0GWVlZUCgUkCQJ58+fx5w5cxwy3u26u7utPxsMBqxevRqhoaGYM2cOCgoKHBb2kiRBqVQCAL799lvrP7yEhATk5eU5ZEwicrMPaC0Wi/XUTXR0NAoLC/Hll19i165dDg17pVIJHx8f+Pn5ISYmxnoqYejQoX3efcteNmzYgLi4OOzZsweBgYFISkrC0KFDkZiYiMTERIeNC9wKXbPZjOvXr0OSJISGhgIA/P39rWHsCCNGjEB5eTkAYNSoUTh//jwA4MqVK/D1dat9DyKv4lbvrvDwcFy8eBGjR48GcCt48vPzsX37dly+fNlh4/r6+qKjowN+fn497qjV1tYGHx/H/T/08fHB7NmzMXXqVOzatQthYWE99rgdqa2tDfn5+ZAkCQqFAs3NzQgPD0d7e7tD/7EuWrQIO3fuxJ49exASEoLVq1cjMjISkZGReP755x02LpHo3OqcfWNjI5RKZY9rsH925swZJCQkOGTcmzdvYsiQIb3mt7a2orm5GSNHjnTIuL908uRJnDlzBvPnz3fKeH3p6OhAS0sLoqOjHTrOjRs3cO3aNVgsFkRERPT5Oyci+3GrsCciIsdwq3P2RETkGAx7IiIBMOyJiATAsCciEgDDnohIAP8PLPehm2TJ0A4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "titanic.hist(by=\"who\", column=\"age\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b0bfe54",
   "metadata": {},
   "source": [
    "There are still one error in the <font color=\"blue\">child</font> group. We have one child, whose age is 25."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "84341fe6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:19:16.124007Z",
     "start_time": "2022-05-03T03:19:16.104146Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>survived</th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>embarked</th>\n",
       "      <th>class</th>\n",
       "      <th>who</th>\n",
       "      <th>adult_male</th>\n",
       "      <th>deck</th>\n",
       "      <th>embark_town</th>\n",
       "      <th>alive</th>\n",
       "      <th>alone</th>\n",
       "      <th>title</th>\n",
       "      <th>lastName</th>\n",
       "      <th>firstName</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>813</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>25.0</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>31.275</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>child</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Miss</td>\n",
       "      <td>Andersson</td>\n",
       "      <td>Ebba Iris Alfrida</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     survived  pclass     sex   age  sibsp  parch    fare embarked  class  \\\n",
       "813         0       3  female  25.0      4      2  31.275        S  Third   \n",
       "\n",
       "       who  adult_male deck  embark_town alive  alone title   lastName  \\\n",
       "813  child       False  NaN  Southampton    no  False  Miss  Andersson   \n",
       "\n",
       "             firstName  \n",
       "813  Ebba Iris Alfrida  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic[(titanic.age > 18 ) & (titanic.who=='child')]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04472d53",
   "metadata": {},
   "source": [
    "In this case, we have to use the value of <font color=\"blue\">sex</font> to figure out the value of <font color=\"blue\">who</font>. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "182aeb93",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:19:40.885769Z",
     "start_time": "2022-05-03T03:19:40.874937Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "survived                       0\n",
       "pclass                         3\n",
       "sex                       female\n",
       "age                         25.0\n",
       "sibsp                          4\n",
       "parch                          2\n",
       "fare                      31.275\n",
       "embarked                       S\n",
       "class                      Third\n",
       "who                        woman\n",
       "adult_male                 False\n",
       "deck                         NaN\n",
       "embark_town          Southampton\n",
       "alive                         no\n",
       "alone                      False\n",
       "title                       Miss\n",
       "lastName               Andersson\n",
       "firstName      Ebba Iris Alfrida\n",
       "Name: 813, dtype: object"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic.loc[(titanic.age == 25) & (titanic.who=='child'), 'who'] = \"woman\"\n",
    "titanic.loc[813,]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "017d3b27",
   "metadata": {},
   "source": [
    "Now, let's compute a simple cross-tabulation of two factors, i.e., <font color=\"blue\">sex</font> and <font color=\"blue\">who</font>:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "8998a77e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:20:09.850993Z",
     "start_time": "2022-05-03T03:20:09.826094Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>sex</th>\n",
       "      <th>female</th>\n",
       "      <th>male</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>who</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>child</th>\n",
       "      <td>54</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>man</th>\n",
       "      <td>0</td>\n",
       "      <td>520</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>woman</th>\n",
       "      <td>260</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "sex    female  male\n",
       "who                \n",
       "child      54    58\n",
       "man         0   520\n",
       "woman     260     0"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.crosstab(titanic[\"who\"],titanic[\"sex\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29a7190f",
   "metadata": {},
   "source": [
    "The tabulation shows that all the women have gender \"female\" and all the men have gender \"male\"."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2793f67c",
   "metadata": {},
   "source": [
    "#### Are the values of \"adult_male\" consistent with the values of \"sex\" and \"who\"?\n",
    "\n",
    "Now, let's look at <font color=\"blue\">adult_male</font>, the value of which should \"True\" if a passenger is male and adult, \"False\" if a passenger is female or a male child. In other words, we need to check the consistency among three variables. What should we do? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "55875601",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:21:15.603909Z",
     "start_time": "2022-05-03T03:21:15.576663Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>survived</th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>embarked</th>\n",
       "      <th>class</th>\n",
       "      <th>who</th>\n",
       "      <th>adult_male</th>\n",
       "      <th>deck</th>\n",
       "      <th>embark_town</th>\n",
       "      <th>alive</th>\n",
       "      <th>alone</th>\n",
       "      <th>title</th>\n",
       "      <th>lastName</th>\n",
       "      <th>firstName</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>34.3750</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>child</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Ford</td>\n",
       "      <td>William Neal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>138</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9.2167</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>child</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Osen</td>\n",
       "      <td>Olaf Elon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>163</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>child</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Calic</td>\n",
       "      <td>Jovo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>220</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>child</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Sunderland</td>\n",
       "      <td>Victor Francis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>266</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>39.6875</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>child</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Panula</td>\n",
       "      <td>Ernesti Arvid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>282</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9.5000</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>child</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>de Pelsmaeker</td>\n",
       "      <td>Alfons</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>333</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>18.0000</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>child</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Vander Planke</td>\n",
       "      <td>Leo Edmondus</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>433</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.1250</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>child</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Kallio</td>\n",
       "      <td>Nikolai Erland</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>500</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>child</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Calic</td>\n",
       "      <td>Petar</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>532</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>17.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>7.2292</td>\n",
       "      <td>C</td>\n",
       "      <td>Third</td>\n",
       "      <td>child</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Cherbourg</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Elias</td>\n",
       "      <td>Joseph Jr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>550</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>110.8833</td>\n",
       "      <td>C</td>\n",
       "      <td>First</td>\n",
       "      <td>child</td>\n",
       "      <td>True</td>\n",
       "      <td>C</td>\n",
       "      <td>Cherbourg</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Thayer</td>\n",
       "      <td>John Borland Jr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>574</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>child</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Rush</td>\n",
       "      <td>Alfred George John</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>721</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>17.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>7.0542</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>child</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Jensen</td>\n",
       "      <td>Svend Lauritz</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>746</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>20.2500</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>child</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Abbott</td>\n",
       "      <td>Rossmore Edward</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>764</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.7750</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>child</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Eklund</td>\n",
       "      <td>Hans Linus</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>791</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>26.0000</td>\n",
       "      <td>S</td>\n",
       "      <td>Second</td>\n",
       "      <td>child</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Gaskell</td>\n",
       "      <td>Alfred</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>841</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10.5000</td>\n",
       "      <td>S</td>\n",
       "      <td>Second</td>\n",
       "      <td>child</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Mudd</td>\n",
       "      <td>Thomas Charles</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>844</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>child</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Culumovic</td>\n",
       "      <td>Jeso</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     survived  pclass   sex   age  sibsp  parch      fare embarked   class  \\\n",
       "86          0       3  male  16.0      1      3   34.3750        S   Third   \n",
       "138         0       3  male  16.0      0      0    9.2167        S   Third   \n",
       "163         0       3  male  17.0      0      0    8.6625        S   Third   \n",
       "220         1       3  male  16.0      0      0    8.0500        S   Third   \n",
       "266         0       3  male  16.0      4      1   39.6875        S   Third   \n",
       "282         0       3  male  16.0      0      0    9.5000        S   Third   \n",
       "333         0       3  male  16.0      2      0   18.0000        S   Third   \n",
       "433         0       3  male  17.0      0      0    7.1250        S   Third   \n",
       "500         0       3  male  17.0      0      0    8.6625        S   Third   \n",
       "532         0       3  male  17.0      1      1    7.2292        C   Third   \n",
       "550         1       1  male  17.0      0      2  110.8833        C   First   \n",
       "574         0       3  male  16.0      0      0    8.0500        S   Third   \n",
       "721         0       3  male  17.0      1      0    7.0542        S   Third   \n",
       "746         0       3  male  16.0      1      1   20.2500        S   Third   \n",
       "764         0       3  male  16.0      0      0    7.7750        S   Third   \n",
       "791         0       2  male  16.0      0      0   26.0000        S  Second   \n",
       "841         0       2  male  16.0      0      0   10.5000        S  Second   \n",
       "844         0       3  male  17.0      0      0    8.6625        S   Third   \n",
       "\n",
       "       who  adult_male deck  embark_town alive  alone title       lastName  \\\n",
       "86   child        True  NaN  Southampton    no  False    Mr           Ford   \n",
       "138  child        True  NaN  Southampton    no   True    Mr           Osen   \n",
       "163  child        True  NaN  Southampton    no   True    Mr          Calic   \n",
       "220  child        True  NaN  Southampton   yes   True    Mr     Sunderland   \n",
       "266  child        True  NaN  Southampton    no  False    Mr         Panula   \n",
       "282  child        True  NaN  Southampton    no   True    Mr  de Pelsmaeker   \n",
       "333  child        True  NaN  Southampton    no  False    Mr  Vander Planke   \n",
       "433  child        True  NaN  Southampton    no   True    Mr         Kallio   \n",
       "500  child        True  NaN  Southampton    no   True    Mr          Calic   \n",
       "532  child        True  NaN    Cherbourg    no  False    Mr          Elias   \n",
       "550  child        True    C    Cherbourg   yes  False    Mr         Thayer   \n",
       "574  child        True  NaN  Southampton    no   True    Mr           Rush   \n",
       "721  child        True  NaN  Southampton    no  False    Mr         Jensen   \n",
       "746  child        True  NaN  Southampton    no  False    Mr         Abbott   \n",
       "764  child        True  NaN  Southampton    no   True    Mr         Eklund   \n",
       "791  child        True  NaN  Southampton    no   True    Mr        Gaskell   \n",
       "841  child        True  NaN  Southampton    no   True    Mr           Mudd   \n",
       "844  child        True  NaN  Southampton    no   True    Mr      Culumovic   \n",
       "\n",
       "              firstName  \n",
       "86         William Neal  \n",
       "138           Olaf Elon  \n",
       "163                Jovo  \n",
       "220      Victor Francis  \n",
       "266       Ernesti Arvid  \n",
       "282              Alfons  \n",
       "333        Leo Edmondus  \n",
       "433      Nikolai Erland  \n",
       "500               Petar  \n",
       "532           Joseph Jr  \n",
       "550     John Borland Jr  \n",
       "574  Alfred George John  \n",
       "721       Svend Lauritz  \n",
       "746     Rossmore Edward  \n",
       "764          Hans Linus  \n",
       "791              Alfred  \n",
       "841      Thomas Charles  \n",
       "844                Jeso  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic[(titanic.adult_male) & ((titanic.sex != \"male\") | (titanic.who != \"man\"))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "d95d8440",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:21:30.597435Z",
     "start_time": "2022-05-03T03:21:30.583495Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>survived</th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>embarked</th>\n",
       "      <th>class</th>\n",
       "      <th>who</th>\n",
       "      <th>adult_male</th>\n",
       "      <th>deck</th>\n",
       "      <th>embark_town</th>\n",
       "      <th>alive</th>\n",
       "      <th>alone</th>\n",
       "      <th>title</th>\n",
       "      <th>lastName</th>\n",
       "      <th>firstName</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [survived, pclass, sex, age, sibsp, parch, fare, embarked, class, who, adult_male, deck, embark_town, alive, alone, title, lastName, firstName]\n",
       "Index: []"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic[(titanic.adult_male == False) & ((titanic.sex == \"male\") & (titanic.age > 18))]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b351830d",
   "metadata": {},
   "source": [
    "According to our assumption on the age of children, we need to change the value of <font color=\"blue\">adult_male</font> form True to False. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "2c86c2cb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:21:55.361443Z",
     "start_time": "2022-05-03T03:21:55.355146Z"
    }
   },
   "outputs": [],
   "source": [
    "titanic.loc[(titanic.adult_male) & ((titanic.sex != \"male\") | (titanic.who != \"man\")), \"adult_male\"]=False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fe5e492",
   "metadata": {},
   "source": [
    "Now, another question is if it is possible for a child less than 10-years old to be on-board the ship alone. Let's have a look"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "d4001981",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:22:11.957001Z",
     "start_time": "2022-05-03T03:22:11.930100Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>survived</th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>embarked</th>\n",
       "      <th>class</th>\n",
       "      <th>who</th>\n",
       "      <th>adult_male</th>\n",
       "      <th>deck</th>\n",
       "      <th>embark_town</th>\n",
       "      <th>alive</th>\n",
       "      <th>alone</th>\n",
       "      <th>title</th>\n",
       "      <th>lastName</th>\n",
       "      <th>firstName</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>777</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12.475</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>child</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>True</td>\n",
       "      <td>Miss</td>\n",
       "      <td>Emanuel</td>\n",
       "      <td>Virginia Ethel</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     survived  pclass     sex  age  sibsp  parch    fare embarked  class  \\\n",
       "777         1       3  female  5.0      0      0  12.475        S  Third   \n",
       "\n",
       "       who  adult_male deck  embark_town alive  alone title lastName  \\\n",
       "777  child       False  NaN  Southampton   yes   True  Miss  Emanuel   \n",
       "\n",
       "          firstName  \n",
       "777  Virginia Ethel  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic[(titanic.age < 10) & (titanic.alone)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ee74595",
   "metadata": {},
   "source": [
    "The output shows that there was a 5-years old girl who was on-board and survived. Should we change the value of <font color=\"blue\">alone</font>? The value is consistent with the value of <font color=\"blue\">silbsp</font> and that of <font color=\"blue\">parch</font>. In this case, we might choose to keep it as it is."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a5268e6",
   "metadata": {},
   "source": [
    "#### Are there any duplicated records?\n",
    "\n",
    "If we assume that <font color=\"blue\">firstName</font>, <font color=\"blue\">lastName</font> and <font color=\"blue\">age</font> can uniquely identify a passenger, we can then use the three values to check whether or not the dataset contains duplicated records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "6debc7dc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:22:55.857178Z",
     "start_time": "2022-05-03T03:22:55.834725Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>survived</th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>embarked</th>\n",
       "      <th>class</th>\n",
       "      <th>who</th>\n",
       "      <th>adult_male</th>\n",
       "      <th>deck</th>\n",
       "      <th>embark_town</th>\n",
       "      <th>alive</th>\n",
       "      <th>alone</th>\n",
       "      <th>title</th>\n",
       "      <th>lastName</th>\n",
       "      <th>firstName</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>889</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>C</td>\n",
       "      <td>First</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>C</td>\n",
       "      <td>Cherbourg</td>\n",
       "      <td>yes</td>\n",
       "      <td>True</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Behr</td>\n",
       "      <td>Karl Howell</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>891</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>Q</td>\n",
       "      <td>First</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>C</td>\n",
       "      <td>Cherbourg</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Behr</td>\n",
       "      <td>Karl Howell</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     survived  pclass   sex   age  sibsp  parch  fare embarked  class  who  \\\n",
       "889         1       1  male  26.0      0      0  30.0        C  First  man   \n",
       "891         1       1  male  26.0      0      0  30.0        Q  First  man   \n",
       "\n",
       "     adult_male deck embark_town alive  alone title lastName    firstName  \n",
       "889        True    C   Cherbourg   yes   True    Mr     Behr  Karl Howell  \n",
       "891        True    C   Cherbourg    no  False    Mr     Behr  Karl Howell  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic[titanic.duplicated([\"firstName\", \"lastName\", \"age\"], keep=False)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d120d97",
   "metadata": {},
   "source": [
    "The output above shows that there are two duplicated records. If you carefully check the two records, the second one contains inconsistent values. For example, <font color=\"blue\">survived</font> = 1, but <font color=\"blue\">alive</font> = no, and <font color=\"blue\">embarked</font> = C, but <font color=\"blue\">embard_town</font> = Cherbourg. Taking into account these two observations, we can choose to remove the second record and just keep the first one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "183c27f6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:23:50.641637Z",
     "start_time": "2022-05-03T03:23:50.622652Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>survived</th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>embarked</th>\n",
       "      <th>class</th>\n",
       "      <th>who</th>\n",
       "      <th>adult_male</th>\n",
       "      <th>deck</th>\n",
       "      <th>embark_town</th>\n",
       "      <th>alive</th>\n",
       "      <th>alone</th>\n",
       "      <th>title</th>\n",
       "      <th>lastName</th>\n",
       "      <th>firstName</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>891</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>Q</td>\n",
       "      <td>First</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>C</td>\n",
       "      <td>Cherbourg</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "      <td>Mr</td>\n",
       "      <td>Behr</td>\n",
       "      <td>Karl Howell</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     survived  pclass   sex   age  sibsp  parch  fare embarked  class  who  \\\n",
       "891         1       1  male  26.0      0      0  30.0        Q  First  man   \n",
       "\n",
       "     adult_male deck embark_town alive  alone title lastName    firstName  \n",
       "891        True    C   Cherbourg    no  False    Mr     Behr  Karl Howell  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic[titanic.duplicated([\"firstName\", \"lastName\", \"age\"], keep=\"first\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "d720bf16",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-03T03:24:08.298775Z",
     "start_time": "2022-05-03T03:24:08.290876Z"
    }
   },
   "outputs": [],
   "source": [
    "titanic.drop_duplicates([\"firstName\", \"lastName\", \"age\"], keep='first', inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12a3e38e",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In this tutorial we demonstrate how to identify and correct some syntactic and semantic data errors. We will cover missing values and outliers separately in the following two weeks' tutorial."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}