DeepLearningProject-Solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final Project\n",
    "\n",
    "## Predict whether a mammogram mass is benign or malignant\n",
    "\n",
    "We'll be using the \"mammographic masses\" public dataset from the UCI repository (source: https://archive.ics.uci.edu/ml/datasets/Mammographic+Mass)\n",
    "\n",
    "This data contains 961 instances of masses detected in mammograms, and contains the following attributes:\n",
    "\n",
    "\n",
    "   1. BI-RADS assessment: 1 to 5 (ordinal)  \n",
    "   2. Age: patient's age in years (integer)\n",
    "   3. Shape: mass shape: round=1 oval=2 lobular=3 irregular=4 (nominal)\n",
    "   4. Margin: mass margin: circumscribed=1 microlobulated=2 obscured=3 ill-defined=4 spiculated=5 (nominal)\n",
    "   5. Density: mass density high=1 iso=2 low=3 fat-containing=4 (ordinal)\n",
    "   6. Severity: benign=0 or malignant=1 (binominal)\n",
    "   \n",
    "BI-RADS is an assesment of how confident the severity classification is; it is not a \"predictive\" attribute and so we will discard it. The age, shape, margin, and density attributes are the features that we will build our model with, and \"severity\" is the classification we will attempt to predict based on those attributes.\n",
    "\n",
    "Although \"shape\" and \"margin\" are nominal data types, which sklearn typically doesn't deal with well, they are close enough to ordinal that we shouldn't just discard them. The \"shape\" for example is ordered increasingly from round to irregular.\n",
    "\n",
    "A lot of unnecessary anguish and surgery arises from false positives arising from mammogram results. If we can build a better way to interpret them through supervised machine learning, it could improve a lot of lives.\n",
    "\n",
    "## Your assignment\n",
    "\n",
    "Build a Multi-Layer Perceptron and train it to classify masses as benign or malignant based on its features.\n",
    "\n",
    "The data needs to be cleaned; many rows contain missing data, and there may be erroneous data identifiable as outliers as well.\n",
    "\n",
    "Remember to normalize your data first! And experiment with different topologies, optimizers, and hyperparameters.\n",
    "\n",
    "I was able to achieve over 80% accuracy - can you beat that?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's begin: prepare your data\n",
    "\n",
    "Start by importing the mammographic_masses.data.txt file into a Pandas dataframe (hint: use read_csv) and take a look at it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>5</th>\n",
       "      <th>67</th>\n",
       "      <th>3</th>\n",
       "      <th>5.1</th>\n",
       "      <th>3.1</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4</td>\n",
       "      <td>43</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>?</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>58</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>28</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5</td>\n",
       "      <td>74</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>?</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>65</td>\n",
       "      <td>1</td>\n",
       "      <td>?</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   5  67  3 5.1 3.1  1\n",
       "0  4  43  1   1   ?  1\n",
       "1  5  58  4   5   3  1\n",
       "2  4  28  1   1   3  0\n",
       "3  5  74  1   5   ?  1\n",
       "4  4  65  1   ?   3  0"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "masses_data = pd.read_csv('mammographic_masses.data.txt')\n",
    "masses_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure you use the optional parmaters in read_csv to convert missing data (indicated by a ?) into NaN, and to add the appropriate column names (BI_RADS, age, shape, margin, density, and severity):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BI-RADS</th>\n",
       "      <th>age</th>\n",
       "      <th>shape</th>\n",
       "      <th>margin</th>\n",
       "      <th>density</th>\n",
       "      <th>severity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.0</td>\n",
       "      <td>43.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>74.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   BI-RADS   age  shape  margin  density  severity\n",
       "0      5.0  67.0    3.0     5.0      3.0         1\n",
       "1      4.0  43.0    1.0     1.0      NaN         1\n",
       "2      5.0  58.0    4.0     5.0      3.0         1\n",
       "3      4.0  28.0    1.0     1.0      3.0         0\n",
       "4      5.0  74.0    1.0     5.0      NaN         1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "masses_data = pd.read_csv('mammographic_masses.data.txt', na_values=['?'], names = ['BI-RADS', 'age', 'shape', 'margin', 'density', 'severity'])\n",
    "masses_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate whether the data needs cleaning; your model is only as good as the data it's given. Hint: use describe() on the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BI-RADS</th>\n",
       "      <th>age</th>\n",
       "      <th>shape</th>\n",
       "      <th>margin</th>\n",
       "      <th>density</th>\n",
       "      <th>severity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>959.000000</td>\n",
       "      <td>956.000000</td>\n",
       "      <td>930.000000</td>\n",
       "      <td>913.000000</td>\n",
       "      <td>885.000000</td>\n",
       "      <td>961.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>4.348279</td>\n",
       "      <td>55.487448</td>\n",
       "      <td>2.721505</td>\n",
       "      <td>2.796276</td>\n",
       "      <td>2.910734</td>\n",
       "      <td>0.463059</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.783031</td>\n",
       "      <td>14.480131</td>\n",
       "      <td>1.242792</td>\n",
       "      <td>1.566546</td>\n",
       "      <td>0.380444</td>\n",
       "      <td>0.498893</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>18.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>4.000000</td>\n",
       "      <td>45.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>4.000000</td>\n",
       "      <td>57.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>5.000000</td>\n",
       "      <td>66.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>55.000000</td>\n",
       "      <td>96.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          BI-RADS         age       shape      margin     density    severity\n",
       "count  959.000000  956.000000  930.000000  913.000000  885.000000  961.000000\n",
       "mean     4.348279   55.487448    2.721505    2.796276    2.910734    0.463059\n",
       "std      1.783031   14.480131    1.242792    1.566546    0.380444    0.498893\n",
       "min      0.000000   18.000000    1.000000    1.000000    1.000000    0.000000\n",
       "25%      4.000000   45.000000    2.000000    1.000000    3.000000    0.000000\n",
       "50%      4.000000   57.000000    3.000000    3.000000    3.000000    0.000000\n",
       "75%      5.000000   66.000000    4.000000    4.000000    3.000000    1.000000\n",
       "max     55.000000   96.000000    4.000000    5.000000    4.000000    1.000000"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "masses_data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are quite a few missing values in the data set. Before we just drop every row that's missing data, let's make sure we don't bias our data in doing so. Does there appear to be any sort of correlation to what sort of data has missing fields? If there were, we'd have to try and go back and fill that data in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BI-RADS</th>\n",
       "      <th>age</th>\n",
       "      <th>shape</th>\n",
       "      <th>margin</th>\n",
       "      <th>density</th>\n",
       "      <th>severity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.0</td>\n",
       "      <td>43.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>74.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>4.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>5.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>5.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>4.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>4.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>NaN</td>\n",
       "      <td>66.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>4.0</td>\n",
       "      <td>43.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>2.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>5.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>4.0</td>\n",
       "      <td>77.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>4.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>4.0</td>\n",
       "      <td>59.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>4.0</td>\n",
       "      <td>61.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>5.0</td>\n",
       "      <td>44.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>4.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>2.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>4.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>4.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>4.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>4.0</td>\n",
       "      <td>43.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>5.0</td>\n",
       "      <td>51.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>4.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>5.0</td>\n",
       "      <td>59.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>3.0</td>\n",
       "      <td>57.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>4.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>5.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>5.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>496</th>\n",
       "      <td>4.0</td>\n",
       "      <td>82.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>501</th>\n",
       "      <td>5.0</td>\n",
       "      <td>59.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>519</th>\n",
       "      <td>3.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>520</th>\n",
       "      <td>4.0</td>\n",
       "      <td>62.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>521</th>\n",
       "      <td>5.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>531</th>\n",
       "      <td>4.0</td>\n",
       "      <td>55.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>537</th>\n",
       "      <td>5.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>541</th>\n",
       "      <td>4.0</td>\n",
       "      <td>49.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>554</th>\n",
       "      <td>5.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>561</th>\n",
       "      <td>2.0</td>\n",
       "      <td>59.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>569</th>\n",
       "      <td>4.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>574</th>\n",
       "      <td>4.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>581</th>\n",
       "      <td>2.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>614</th>\n",
       "      <td>3.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>627</th>\n",
       "      <td>4.0</td>\n",
       "      <td>57.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>660</th>\n",
       "      <td>4.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>661</th>\n",
       "      <td>4.0</td>\n",
       "      <td>51.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>662</th>\n",
       "      <td>3.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>665</th>\n",
       "      <td>4.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>677</th>\n",
       "      <td>4.0</td>\n",
       "      <td>57.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>683</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>691</th>\n",
       "      <td>4.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>723</th>\n",
       "      <td>4.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>745</th>\n",
       "      <td>6.0</td>\n",
       "      <td>76.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>752</th>\n",
       "      <td>5.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>778</th>\n",
       "      <td>4.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>819</th>\n",
       "      <td>4.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>824</th>\n",
       "      <td>6.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>884</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>923</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>130 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     BI-RADS   age  shape  margin  density  severity\n",
       "1        4.0  43.0    1.0     1.0      NaN         1\n",
       "4        5.0  74.0    1.0     5.0      NaN         1\n",
       "5        4.0  65.0    1.0     NaN      3.0         0\n",
       "6        4.0  70.0    NaN     NaN      3.0         0\n",
       "7        5.0  42.0    1.0     NaN      3.0         0\n",
       "9        5.0  60.0    NaN     5.0      1.0         1\n",
       "12       4.0  64.0    1.0     NaN      3.0         0\n",
       "19       4.0  40.0    1.0     NaN      NaN         0\n",
       "20       NaN  66.0    NaN     NaN      1.0         1\n",
       "22       4.0  43.0    1.0     NaN      NaN         0\n",
       "26       2.0  66.0    1.0     1.0      NaN         0\n",
       "27       5.0  63.0    3.0     NaN      3.0         0\n",
       "35       4.0  77.0    3.0     NaN      NaN         0\n",
       "38       4.0  48.0    4.0     5.0      NaN         1\n",
       "40       4.0  59.0    2.0     1.0      NaN         0\n",
       "43       4.0  61.0    2.0     1.0      NaN         0\n",
       "45       5.0  44.0    2.0     4.0      NaN         1\n",
       "47       4.0  23.0    1.0     1.0      NaN         0\n",
       "48       2.0  42.0    NaN     NaN      4.0         0\n",
       "52       4.0  23.0    1.0     1.0      NaN         0\n",
       "53       4.0  63.0    2.0     1.0      NaN         0\n",
       "54       4.0  53.0    NaN     5.0      3.0         1\n",
       "55       4.0  43.0    3.0     4.0      NaN         0\n",
       "57       5.0  51.0    2.0     4.0      NaN         0\n",
       "58       4.0  45.0    2.0     1.0      NaN         0\n",
       "59       5.0  59.0    2.0     NaN      NaN         1\n",
       "63       3.0  57.0    2.0     1.0      NaN         0\n",
       "65       4.0  25.0    2.0     1.0      NaN         0\n",
       "67       5.0  72.0    4.0     3.0      NaN         1\n",
       "74       5.0  70.0    NaN     4.0      NaN         1\n",
       "..       ...   ...    ...     ...      ...       ...\n",
       "496      4.0  82.0    NaN     5.0      3.0         1\n",
       "501      5.0  59.0    4.0     4.0      NaN         1\n",
       "519      3.0  68.0    NaN     NaN      3.0         0\n",
       "520      4.0  62.0    4.0     NaN      3.0         1\n",
       "521      5.0  65.0    1.0     NaN      3.0         1\n",
       "531      4.0  55.0    NaN     NaN      3.0         0\n",
       "537      5.0  63.0    NaN     4.0      3.0         1\n",
       "541      4.0  49.0    2.0     NaN      3.0         0\n",
       "554      5.0  70.0    NaN     5.0      3.0         1\n",
       "561      2.0  59.0    NaN     4.0      3.0         0\n",
       "569      4.0  64.0    3.0     4.0      NaN         1\n",
       "574      4.0  60.0    3.0     NaN      NaN         0\n",
       "581      2.0  65.0    NaN     1.0      2.0         0\n",
       "614      3.0  46.0    NaN     5.0      NaN         1\n",
       "627      4.0  57.0    2.0     1.0      NaN         0\n",
       "660      4.0  58.0    NaN     4.0      3.0         1\n",
       "661      4.0  51.0    NaN     4.0      3.0         0\n",
       "662      3.0  50.0    NaN     NaN      3.0         1\n",
       "665      4.0  27.0    2.0     1.0      NaN         0\n",
       "677      4.0  57.0    4.0     4.0      NaN         1\n",
       "683      5.0   NaN    3.0     3.0      3.0         1\n",
       "691      4.0  72.0    3.0     NaN      3.0         0\n",
       "723      4.0  60.0    3.0     NaN      4.0         0\n",
       "745      6.0  76.0    3.0     NaN      3.0         0\n",
       "752      5.0  48.0    NaN     4.0      NaN         1\n",
       "778      4.0  60.0    NaN     4.0      3.0         0\n",
       "819      4.0  35.0    3.0     NaN      2.0         0\n",
       "824      6.0  40.0    NaN     3.0      4.0         1\n",
       "884      5.0   NaN    4.0     4.0      3.0         1\n",
       "923      5.0   NaN    4.0     3.0      3.0         1\n",
       "\n",
       "[130 rows x 6 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "masses_data.loc[(masses_data['age'].isnull()) |\n",
    "              (masses_data['shape'].isnull()) |\n",
    "              (masses_data['margin'].isnull()) |\n",
    "              (masses_data['density'].isnull())]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the missing data seems randomly distributed, go ahead and drop rows with missing data. Hint: use dropna()."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BI-RADS</th>\n",
       "      <th>age</th>\n",
       "      <th>shape</th>\n",
       "      <th>margin</th>\n",
       "      <th>density</th>\n",
       "      <th>severity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>830.000000</td>\n",
       "      <td>830.000000</td>\n",
       "      <td>830.000000</td>\n",
       "      <td>830.000000</td>\n",
       "      <td>830.000000</td>\n",
       "      <td>830.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>4.393976</td>\n",
       "      <td>55.781928</td>\n",
       "      <td>2.781928</td>\n",
       "      <td>2.813253</td>\n",
       "      <td>2.915663</td>\n",
       "      <td>0.485542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.888371</td>\n",
       "      <td>14.671782</td>\n",
       "      <td>1.242361</td>\n",
       "      <td>1.567175</td>\n",
       "      <td>0.350936</td>\n",
       "      <td>0.500092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>18.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>4.000000</td>\n",
       "      <td>46.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>4.000000</td>\n",
       "      <td>57.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>5.000000</td>\n",
       "      <td>66.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>55.000000</td>\n",
       "      <td>96.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          BI-RADS         age       shape      margin     density    severity\n",
       "count  830.000000  830.000000  830.000000  830.000000  830.000000  830.000000\n",
       "mean     4.393976   55.781928    2.781928    2.813253    2.915663    0.485542\n",
       "std      1.888371   14.671782    1.242361    1.567175    0.350936    0.500092\n",
       "min      0.000000   18.000000    1.000000    1.000000    1.000000    0.000000\n",
       "25%      4.000000   46.000000    2.000000    1.000000    3.000000    0.000000\n",
       "50%      4.000000   57.000000    3.000000    3.000000    3.000000    0.000000\n",
       "75%      5.000000   66.000000    4.000000    4.000000    3.000000    1.000000\n",
       "max     55.000000   96.000000    4.000000    5.000000    4.000000    1.000000"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "masses_data.dropna(inplace=True)\n",
    "masses_data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Next you'll need to convert the Pandas dataframes into numpy arrays that can be used by scikit_learn. Create an array that extracts only the feature data we want to work with (age, shape, margin, and density) and another array that contains the classes (severity). You'll also need an array of the feature name labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[67.,  3.,  5.,  3.],\n",
       "       [58.,  4.,  5.,  3.],\n",
       "       [28.,  1.,  1.,  3.],\n",
       "       ...,\n",
       "       [64.,  4.,  5.,  3.],\n",
       "       [66.,  4.,  5.,  3.],\n",
       "       [62.,  3.,  3.,  3.]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_features = masses_data[['age', 'shape',\n",
    "                             'margin', 'density']].values\n",
    "\n",
    "\n",
    "all_classes = masses_data['severity'].values\n",
    "\n",
    "feature_names = ['age', 'shape', 'margin', 'density']\n",
    "\n",
    "all_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of our models require the input data to be normalized, so go ahead and normalize the attribute data. Hint: use preprocessing.StandardScaler()."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.7650629 ,  0.17563638,  1.39618483,  0.24046607],\n",
       "       [ 0.15127063,  0.98104077,  1.39618483,  0.24046607],\n",
       "       [-1.89470363, -1.43517241, -1.157718  ,  0.24046607],\n",
       "       ...,\n",
       "       [ 0.56046548,  0.98104077,  1.39618483,  0.24046607],\n",
       "       [ 0.69686376,  0.98104077,  1.39618483,  0.24046607],\n",
       "       [ 0.42406719,  0.17563638,  0.11923341,  0.24046607]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import preprocessing\n",
    "\n",
    "scaler = preprocessing.StandardScaler()\n",
    "all_features_scaled = scaler.fit_transform(all_features)\n",
    "all_features_scaled"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now set up an actual MLP model using Keras:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras.models import Sequential\n",
    "\n",
    "def create_model():\n",
    "    model = Sequential()\n",
    "    #4 feature inputs going into an 6-unit layer (more does not seem to help - in fact you can go down to 4)\n",
    "    model.add(Dense(6, input_dim=4, kernel_initializer='normal', activation='relu'))\n",
    "    # \"Deep learning\" turns out to be unnecessary - this additional hidden layer doesn't help either.\n",
    "    #model.add(Dense(4, kernel_initializer='normal', activation='relu'))\n",
    "    # Output layer with a binary classification (benign or malignant)\n",
    "    model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))\n",
    "    # Compile model; rmsprop seemed to work best\n",
    "    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8072289168834687"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "from tensorflow.keras.wrappers.scikit_learn import KerasClassifier\n",
    "\n",
    "# Wrap our Keras model in an estimator compatible with scikit_learn\n",
    "estimator = KerasClassifier(build_fn=create_model, epochs=100, verbose=0)\n",
    "# Now we can use scikit_learn's cross_val_score to evaluate this model identically to the others\n",
    "cv_scores = cross_val_score(estimator, all_features_scaled, all_classes, cv=10)\n",
    "cv_scores.mean()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## How did you do?\n",
    "\n",
    "Which topology, and which choice of hyperparameters, performed the best? Feel free to share your results!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}