Keras Basic Sequential Application to the MNIST Dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Goal:**\n",
    "Test out a quick artificial neural network model for the MNIST dataset using a sequential model to construct a neural network that can classify the handwritten digits for the MNIST dataset. Technically for a neural network to be considered an implementation of Deep Learning there has to be roughly 3 or more node layers, so this application uses two sequential models; one that is considered Deep Learning and one that is not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# deep learning imports\n",
    "import keras\n",
    "from keras.layers import Dense\n",
    "from keras.models import Sequential\n",
    "from tensorflow.keras.utils import to_categorical # makes one-hot encoding easy\n",
    "from keras.callbacks import EarlyStopping\n",
    "\n",
    "# notebook settings\n",
    "%matplotlib inline\n",
    "pd.options.display.max_columns = 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data could have been loaded directly through keras. I chose not to do this as it is important to remember that the fit function here requires a one-hot encoded target set which the \"to_categorical\" function helps to achieve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['data', 'target', 'frame', 'categories', 'feature_names', 'target_names', 'DESCR', 'details', 'url'])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load the MNIST dataset\n",
    "from sklearn.datasets import fetch_openml\n",
    "mnist = fetch_openml('mnist_784', version=1)\n",
    "mnist.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 0., ..., 0., 0., 0.],\n",
       "       [1., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       ...,\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.]], dtype=float32)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# set X and target matrices\n",
    "X, y = mnist['data'], mnist['target'].astype('int')\n",
    "\n",
    "target = to_categorical(y)\n",
    "target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like when training any other model, we use a validation set to determine how well the model will perform on unseen data. With keras, that option is built in via the \"validation_split\" parameter. The trouble here is that there is not a way to stratify the validation sample which could affect model performance if there is a noticeable imbalance in the data. The following checks to see if there is an imbalance prior to running the model: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    0.112529\n",
       "7    0.104186\n",
       "3    0.102014\n",
       "2    0.099857\n",
       "9    0.099400\n",
       "0    0.098614\n",
       "6    0.098229\n",
       "8    0.097500\n",
       "4    0.097486\n",
       "5    0.090186\n",
       "dtype: float64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(y).value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The proportions are roughly the same so stratified samples are not necessary.\n",
    "\n",
    "Typically Deep Learning is applied when large amounts of training data are available, however it performs surprisingly well when trained on only 20% of the data and predicting on the other 80% as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "438/438 [==============================] - 2s 4ms/step - loss: 4.0522 - accuracy: 0.6783 - val_loss: 1.0826 - val_accuracy: 0.7819\n",
      "Epoch 2/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.7459 - accuracy: 0.8286 - val_loss: 0.7163 - val_accuracy: 0.8392\n",
      "Epoch 3/30\n",
      "438/438 [==============================] - 1s 2ms/step - loss: 0.4999 - accuracy: 0.8717 - val_loss: 0.6722 - val_accuracy: 0.8590\n",
      "Epoch 4/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.4343 - accuracy: 0.8898 - val_loss: 0.5861 - val_accuracy: 0.8728\n",
      "Epoch 5/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.3864 - accuracy: 0.8996 - val_loss: 0.6141 - val_accuracy: 0.8665\n",
      "Epoch 6/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.3470 - accuracy: 0.9100 - val_loss: 0.5392 - val_accuracy: 0.8885\n",
      "Epoch 7/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.3387 - accuracy: 0.9149 - val_loss: 0.5091 - val_accuracy: 0.8859\n",
      "Epoch 8/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.3071 - accuracy: 0.9206 - val_loss: 0.5500 - val_accuracy: 0.8964\n",
      "Epoch 9/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.2875 - accuracy: 0.9264 - val_loss: 0.4927 - val_accuracy: 0.8952\n",
      "Epoch 10/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.2649 - accuracy: 0.9287 - val_loss: 0.4965 - val_accuracy: 0.9021\n",
      "Epoch 11/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.2499 - accuracy: 0.9359 - val_loss: 0.4517 - val_accuracy: 0.9078\n",
      "Epoch 12/30\n",
      "438/438 [==============================] - 1s 2ms/step - loss: 0.2457 - accuracy: 0.9373 - val_loss: 0.4484 - val_accuracy: 0.9119\n",
      "Epoch 13/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.2333 - accuracy: 0.9389 - val_loss: 0.4938 - val_accuracy: 0.9063\n",
      "Epoch 14/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.2147 - accuracy: 0.9423 - val_loss: 0.4618 - val_accuracy: 0.9108\n",
      "Epoch 15/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.2148 - accuracy: 0.9439 - val_loss: 0.4789 - val_accuracy: 0.9107\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1584ba580>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# attempt to train on 20% of the daa and then predict on the rest\n",
    "\n",
    "# build the model\n",
    "model = Sequential()\n",
    "\n",
    "# add early stopping \n",
    "early_stopping_monitor = EarlyStopping(patience=3)\n",
    "\n",
    "# add the layers - rough guess to start\n",
    "model.add(Dense(50, activation='relu', input_shape=(len(mnist['feature_names']),))) #minimum is zero so reLU would work nicely here\n",
    "model.add(Dense(50, activation='relu'))\n",
    "\n",
    "# add final layer\n",
    "model.add(Dense(10, activation='softmax'))\n",
    "\n",
    "# compile\n",
    "model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "\n",
    "# fit and validate\n",
    "model.fit(X, target, validation_split=.80, epochs=30, callbacks=[early_stopping_monitor])\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model shows a validation accuracy of 91% for a 20/80 split of the data which is really good for a basic model trained on only 20% of the data. The following model shows the improvement from using a more proper split of the data (75/25)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "1641/1641 [==============================] - 3s 1ms/step - loss: 1.6087 - accuracy: 0.8008 - val_loss: 0.4826 - val_accuracy: 0.8755\n",
      "Epoch 2/30\n",
      "1641/1641 [==============================] - 2s 1ms/step - loss: 0.4124 - accuracy: 0.8920 - val_loss: 0.3701 - val_accuracy: 0.9003\n",
      "Epoch 3/30\n",
      "1641/1641 [==============================] - 2s 996us/step - loss: 0.3226 - accuracy: 0.9147 - val_loss: 0.3094 - val_accuracy: 0.9249\n",
      "Epoch 4/30\n",
      "1641/1641 [==============================] - 2s 947us/step - loss: 0.2738 - accuracy: 0.9275 - val_loss: 0.2628 - val_accuracy: 0.9325\n",
      "Epoch 5/30\n",
      "1641/1641 [==============================] - 2s 939us/step - loss: 0.2375 - accuracy: 0.9363 - val_loss: 0.2737 - val_accuracy: 0.9302\n",
      "Epoch 6/30\n",
      "1641/1641 [==============================] - 2s 947us/step - loss: 0.2073 - accuracy: 0.9426 - val_loss: 0.2189 - val_accuracy: 0.9439\n",
      "Epoch 7/30\n",
      "1641/1641 [==============================] - 2s 939us/step - loss: 0.1852 - accuracy: 0.9471 - val_loss: 0.1959 - val_accuracy: 0.9470\n",
      "Epoch 8/30\n",
      "1641/1641 [==============================] - 2s 962us/step - loss: 0.1640 - accuracy: 0.9539 - val_loss: 0.2099 - val_accuracy: 0.9470\n",
      "Epoch 9/30\n",
      "1641/1641 [==============================] - 2s 956us/step - loss: 0.1566 - accuracy: 0.9548 - val_loss: 0.2261 - val_accuracy: 0.9403\n",
      "Epoch 10/30\n",
      "1641/1641 [==============================] - 2s 943us/step - loss: 0.1447 - accuracy: 0.9573 - val_loss: 0.1803 - val_accuracy: 0.9519\n",
      "Epoch 11/30\n",
      "1641/1641 [==============================] - 2s 949us/step - loss: 0.1371 - accuracy: 0.9602 - val_loss: 0.1784 - val_accuracy: 0.9554\n",
      "Epoch 12/30\n",
      "1641/1641 [==============================] - 2s 943us/step - loss: 0.1286 - accuracy: 0.9630 - val_loss: 0.1771 - val_accuracy: 0.9580\n",
      "Epoch 13/30\n",
      "1641/1641 [==============================] - 2s 943us/step - loss: 0.1181 - accuracy: 0.9659 - val_loss: 0.1699 - val_accuracy: 0.9579\n",
      "Epoch 14/30\n",
      "1641/1641 [==============================] - 2s 941us/step - loss: 0.1171 - accuracy: 0.9658 - val_loss: 0.1897 - val_accuracy: 0.9527\n",
      "Epoch 15/30\n",
      "1641/1641 [==============================] - 2s 941us/step - loss: 0.1107 - accuracy: 0.9677 - val_loss: 0.1809 - val_accuracy: 0.9565\n",
      "Epoch 16/30\n",
      "1641/1641 [==============================] - 2s 954us/step - loss: 0.1080 - accuracy: 0.9683 - val_loss: 0.1938 - val_accuracy: 0.9535\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x15859e550>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# attempt to train on 20% of the data and then predict on the rest\n",
    "\n",
    "# build the model\n",
    "model = Sequential()\n",
    "\n",
    "# add early stopping \n",
    "early_stopping_monitor = EarlyStopping(patience=3)\n",
    "\n",
    "# add the layers - rough guess to start\n",
    "model.add(Dense(50, activation='relu', input_shape=(len(mnist['feature_names']),))) #minimum is zero so reLU would work nicely here\n",
    "model.add(Dense(50, activation='relu'))\n",
    "\n",
    "# add final layer\n",
    "model.add(Dense(10, activation='softmax'))\n",
    "\n",
    "# compile\n",
    "model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "\n",
    "# fit and validate\n",
    "model.fit(X, target, validation_split=.25, epochs=30, callbacks=[early_stopping_monitor])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model shows a validation accuracy of 95% for a 75/25 split of the data which is amazing for a basic model trained on only 20% of the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the previous is a neural network, there are not enough layers for it to be considered a Deep Learning model. The following takes the model and increase the layers and nodes per layer to see if this enhances the accuracy of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "438/438 [==============================] - 3s 6ms/step - loss: 1.8992 - accuracy: 0.7495 - val_loss: 0.5820 - val_accuracy: 0.8463\n",
      "Epoch 2/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.4515 - accuracy: 0.8826 - val_loss: 0.4788 - val_accuracy: 0.8703\n",
      "Epoch 3/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.2898 - accuracy: 0.9204 - val_loss: 0.3651 - val_accuracy: 0.9042\n",
      "Epoch 4/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.2283 - accuracy: 0.9338 - val_loss: 0.3638 - val_accuracy: 0.9074\n",
      "Epoch 5/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.1960 - accuracy: 0.9449 - val_loss: 0.3315 - val_accuracy: 0.9210\n",
      "Epoch 6/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.1760 - accuracy: 0.9491 - val_loss: 0.3146 - val_accuracy: 0.9246\n",
      "Epoch 7/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.1677 - accuracy: 0.9524 - val_loss: 0.3561 - val_accuracy: 0.9152\n",
      "Epoch 8/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.1326 - accuracy: 0.9599 - val_loss: 0.3325 - val_accuracy: 0.9245\n",
      "Epoch 9/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.1302 - accuracy: 0.9614 - val_loss: 0.3080 - val_accuracy: 0.9281\n",
      "Epoch 10/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.1290 - accuracy: 0.9617 - val_loss: 0.3315 - val_accuracy: 0.9302\n",
      "Epoch 11/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.1156 - accuracy: 0.9671 - val_loss: 0.3399 - val_accuracy: 0.9272\n",
      "Epoch 12/30\n",
      "438/438 [==============================] - 1s 3ms/step - loss: 0.1336 - accuracy: 0.9636 - val_loss: 0.3279 - val_accuracy: 0.9288\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1586cb6a0>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# build the model\n",
    "model2 = Sequential()\n",
    "\n",
    "# add the layers - rough guess to start\n",
    "model2.add(Dense(75, activation='relu', input_shape=(len(mnist['feature_names']),))) #minimum is zero so reLU would work nicely here\n",
    "model2.add(Dense(75, activation='relu'))\n",
    "model2.add(Dense(75, activation='relu'))\n",
    "model2.add(Dense(75, activation='relu'))\n",
    "\n",
    "# add final layer\n",
    "model2.add(Dense(10, activation='softmax'))\n",
    "\n",
    "# compile\n",
    "model2.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "\n",
    "# fit and validate\n",
    "model2.fit(X, target, validation_split=.80, epochs=30, callbacks=[early_stopping_monitor])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "1641/1641 [==============================] - 3s 1ms/step - loss: 0.9397 - accuracy: 0.8392 - val_loss: 0.2792 - val_accuracy: 0.9207\n",
      "Epoch 2/30\n",
      "1641/1641 [==============================] - 2s 1ms/step - loss: 0.2502 - accuracy: 0.9294 - val_loss: 0.2162 - val_accuracy: 0.9377\n",
      "Epoch 3/30\n",
      "1641/1641 [==============================] - 2s 1ms/step - loss: 0.1915 - accuracy: 0.9462 - val_loss: 0.1939 - val_accuracy: 0.9449\n",
      "Epoch 4/30\n",
      "1641/1641 [==============================] - 2s 1ms/step - loss: 0.1657 - accuracy: 0.9527 - val_loss: 0.1580 - val_accuracy: 0.9543\n",
      "Epoch 5/30\n",
      "1641/1641 [==============================] - 2s 1ms/step - loss: 0.1402 - accuracy: 0.9598 - val_loss: 0.1639 - val_accuracy: 0.9551\n",
      "Epoch 6/30\n",
      "1641/1641 [==============================] - 2s 1ms/step - loss: 0.1246 - accuracy: 0.9638 - val_loss: 0.1512 - val_accuracy: 0.9583\n",
      "Epoch 7/30\n",
      "1641/1641 [==============================] - 2s 1ms/step - loss: 0.1147 - accuracy: 0.9672 - val_loss: 0.1441 - val_accuracy: 0.9610\n",
      "Epoch 8/30\n",
      "1641/1641 [==============================] - 2s 1ms/step - loss: 0.0992 - accuracy: 0.9715 - val_loss: 0.1457 - val_accuracy: 0.9629\n",
      "Epoch 9/30\n",
      "1641/1641 [==============================] - 2s 1ms/step - loss: 0.0908 - accuracy: 0.9741 - val_loss: 0.1347 - val_accuracy: 0.9657\n",
      "Epoch 10/30\n",
      "1641/1641 [==============================] - 2s 1ms/step - loss: 0.0823 - accuracy: 0.9762 - val_loss: 0.1264 - val_accuracy: 0.9686\n",
      "Epoch 11/30\n",
      "1641/1641 [==============================] - 2s 1ms/step - loss: 0.0776 - accuracy: 0.9783 - val_loss: 0.1556 - val_accuracy: 0.9629\n",
      "Epoch 12/30\n",
      "1641/1641 [==============================] - 2s 1ms/step - loss: 0.0753 - accuracy: 0.9790 - val_loss: 0.1366 - val_accuracy: 0.9682\n",
      "Epoch 13/30\n",
      "1641/1641 [==============================] - 2s 1ms/step - loss: 0.0672 - accuracy: 0.9814 - val_loss: 0.1320 - val_accuracy: 0.9704\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1589743a0>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# build the model\n",
    "model2 = Sequential()\n",
    "\n",
    "# add the layers - rough guess to start\n",
    "model2.add(Dense(75, activation='relu', input_shape=(len(mnist['feature_names']),))) #minimum is zero so reLU would work nicely here\n",
    "model2.add(Dense(75, activation='relu'))\n",
    "model2.add(Dense(75, activation='relu'))\n",
    "model2.add(Dense(75, activation='relu'))\n",
    "\n",
    "# add final layer\n",
    "model2.add(Dense(10, activation='softmax'))\n",
    "\n",
    "# compile\n",
    "model2.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "\n",
    "# fit and validate\n",
    "model2.fit(X, target, validation_split=.25, epochs=30, callbacks=[early_stopping_monitor])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The new model while only increasing to 92% when training on 20% of the data, increased to 97% accuracy when training on a proper split (75/25)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}