emotion_detection_CV.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Emotion Detection Model generation\n",
    "\n",
    "In this notebook i will explain how to create Convolutional Neural Network Model from scratch by using **Keras: the python deep learning api**. To understand this thouroughly you should have some basic knowledge of python, Convolutional Neural Network(CNN) and the different layers which are used in CNN.\n",
    "\n",
    "\n",
    "I have divided this notebook into tasks to make it easier to understand.\n",
    "\n",
    "## Task 1:\n",
    "\n",
    "Import the required modules that are needed in this project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import keras\n",
    "from keras.preprocessing.image import ImageDataGenerator\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense,Dropout,Activation,Flatten,BatchNormalization\n",
    "from keras.layers import Conv2D,MaxPooling2D\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's define some variables that will save the time of writing the values manually again and again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_classes=5\n",
    "img_rows,img_cols=48,48\n",
    "batch_size=32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The discription for above variables are as follows:\n",
    "\n",
    "- num_classses = 5 : This variable defines the number of classes or the emotions that we will be dealing with in training our model.\n",
    "\n",
    "- img_rows,img_cols=48,48 : These variables define the size of the image array that we will be feeding to our neural network.\n",
    "\n",
    "- batch_size=32: This variable defines the batch size.The batch size is a number of samples processed before the model is updated. The number of epochs is the number of complete passes through the training dataset. The size of a batch must be more than or equal to one and less than or equal to the number of samples in the training dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2:\n",
    "\n",
    "Now time to take out the big guns!!\n",
    "\n",
    "Now it's time to load our dataset. It's the dataset that makes a deeplearning model what it is. Here i am using the **fer2013** dataset which is an open source dataset hosted on [kaggle](https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data). The dataset contains totally 7 classes namely Angry,Disgust,Fear,Happy,Sad,Surprise and Neutral.The training set consists of a total of 28,709 examples.\n",
    "\n",
    "I have segregated the data in different folders containing imgages pertaing to the foldername. For example, Angry folder contains pics with angry faces etc.Here we are using 5 classes which include Angry,Happy,Sad,Surprise and Neutral.So in total i am using 24256 images as training data and using 3006 images as validation data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data_dir='fer2013/train'\n",
    "validation_data_dir='fer2013/validation'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above two lines import the validation and training data. The model is trained on the training dataset and the validation datadet is a part of the original dataset which is seperated from it to check the performance of the model on the data that it has never seen before."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 3:\n",
    "\n",
    "Now we will be using Image Augmentation techniques om our dataset. Image data augmentation is a technique that can be used to artificially expand the size of a training dataset by creating modified versions of images in the dataset.The Keras deep learning neural network library provides the capability to fit models using image data augmentation via the ImageDataGenerator class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_datagen = ImageDataGenerator(\n",
    "    rescale=1./255,\n",
    "    rotation_range=30,\n",
    "    shear_range=0.3,\n",
    "    zoom_range=0.3,\n",
    "    width_shift_range=0.4,\n",
    "    height_shift_range=0.4,\n",
    "    horizontal_flip=True,\n",
    "    fill_mode='nearest')\n",
    "\n",
    "validation_datagen = ImageDataGenerator(rescale=1./255)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The train_datagen variable will artificially expand the dataset using the following:\n",
    "\n",
    "- rotation_range: Degree range for random rotations. Here i am using 30$^\\circ$ \n",
    "- shear_range: Shear Intensity (Shear angle in counter-clockwise direction in degrees). Here i am using 0.3 as shear range.\n",
    "- zoom_range: Range for random zoom..Here i am using 0.3 as zoom range.\n",
    "- width_shift_range: This shifts the images by a value across its width.\n",
    "- height_shift_range : This shifts the images by a value across its height.\n",
    "- horizontal_flip: This flips the images horizontally.\n",
    "- fill_mode: This is used to fill in the pixels after making changes to the orientation of the images by the above used methods. Here i am using 'nearest' as the fill mode as i am instructing it to fill the missing pixels in the image with the nearby pixels.\n",
    "\n",
    "Here i am resclaing the validation data and not performing any other augmentaions as i want to check the model with raw data that is different from the data used in the training of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 24256 images belonging to 5 classes.\n",
      "Found 3006 images belonging to 5 classes.\n"
     ]
    }
   ],
   "source": [
    "train_generator = train_datagen.flow_from_directory(\n",
    "                        train_data_dir,\n",
    "                        color_mode='grayscale',\n",
    "                        target_size=(img_rows,img_cols),\n",
    "                        batch_size=batch_size,\n",
    "                        class_mode='categorical',\n",
    "                        shuffle=True)\n",
    "\n",
    "validation_generator = validation_datagen.flow_from_directory(\n",
    "                                validation_data_dir,\n",
    "                                color_mode='grayscale',\n",
    "                                target_size=(img_rows,img_cols),\n",
    "                                batch_size=batch_size,\n",
    "                                class_mode='categorical',\n",
    "                                shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now i am using the **flow_from_directory()** method to load our dataset from the directory which is augmented and stored in the train_generator and validation_generator varianbles.**flow_from_directory()** actually takes the path to a directory & generates batches of augmented data. So here i am giving some options to the method to automatically change the dimention and divide it in the classes so that it is easier to feed in the model.\n",
    "\n",
    "The options given are:\n",
    "\n",
    "- directory: The directory of the dataset.\n",
    "- color_mode: Here i am converting the images to grayscale as i am not interested in the color of the images but only the expressions.\n",
    "- target_size: Convert the images to a uniform size.\n",
    "- batch_size: To make baches of data to train.\n",
    "- class_mode: Here i am using 'categorical' as the class mode as i am categorizing my images into 5 classes.\n",
    "- shuffle: To shuffle the dataset for better training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 4:\n",
    "\n",
    "The dataset modifications is complete and now it's time to make the brain of the model i.e. the CNN Network.\n",
    "\n",
    "So firstly i will define the type of model that i will be using. Here i am using a **Sequential** model which defines that all the layers in the network will be one after the other sequentially."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The network consists of 7 blocks:\n",
    "(**Note:I will be explaning each layer one by one at the last**)\n",
    "- ### Block-1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Conv2D(32,(3,3),padding='same',kernel_initializer='he_normal',input_shape=(img_rows,img_cols,1)))\n",
    "model.add(Activation('elu'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Conv2D(32,(3,3),padding='same',kernel_initializer='he_normal',input_shape=(img_rows,img_cols,1)))\n",
    "model.add(Activation('elu'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(MaxPooling2D(pool_size=(2,2)))\n",
    "model.add(Dropout(0.2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- ### Block-2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Conv2D(64,(3,3),padding='same',kernel_initializer='he_normal'))\n",
    "model.add(Activation('elu'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Conv2D(64,(3,3),padding='same',kernel_initializer='he_normal'))\n",
    "model.add(Activation('elu'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(MaxPooling2D(pool_size=(2,2)))\n",
    "model.add(Dropout(0.2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- ### Block-3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Conv2D(128,(3,3),padding='same',kernel_initializer='he_normal'))\n",
    "model.add(Activation('elu'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Conv2D(128,(3,3),padding='same',kernel_initializer='he_normal'))\n",
    "model.add(Activation('elu'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(MaxPooling2D(pool_size=(2,2)))\n",
    "model.add(Dropout(0.2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- ### Block-4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Conv2D(256,(3,3),padding='same',kernel_initializer='he_normal'))\n",
    "model.add(Activation('elu'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Conv2D(256,(3,3),padding='same',kernel_initializer='he_normal'))\n",
    "model.add(Activation('elu'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(MaxPooling2D(pool_size=(2,2)))\n",
    "model.add(Dropout(0.2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- ### Block-5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Flatten())\n",
    "model.add(Dense(64,kernel_initializer='he_normal'))\n",
    "model.add(Activation('elu'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Dropout(0.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- ### Block-6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Dense(64,kernel_initializer='he_normal'))\n",
    "model.add(Activation('elu'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Dropout(0.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- ### Block-7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Dense(num_classes,kernel_initializer='he_normal'))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This seems a lot actually it is a lot.\n",
    "\n",
    "Here i have used the layers of 7 types which are present in **keras.layers**.\n",
    "\n",
    "The layers are:\n",
    "\n",
    "- Conv2D(\n",
    "    filters, kernel_size, strides=(1, 1), padding='valid', data_format=None,\n",
    "    dilation_rate=(1, 1), activation=None, use_bias=True,\n",
    "    kernel_initializer='glorot_uniform', bias_initializer='zeros',\n",
    "    kernel_regularizer=None, bias_regularizer=None, activity_regularizer=None,\n",
    "    kernel_constraint=None, bias_constraint=None, **kwargs\n",
    ")\n",
    "- Activation(activation_type)\n",
    "\n",
    "- BatchNormalization()\n",
    "\n",
    "- MaxPooling2D(pool_size, strides, padding, data_format, **kwargs)\n",
    "\n",
    "- Dropout(dropout_value) \n",
    "\n",
    "- Flatten()\n",
    "\n",
    "- Dense(\n",
    "    units,\n",
    "    activation=None,\n",
    "    use_bias=True,\n",
    "    kernel_initializer=\"glorot_uniform\",\n",
    "    bias_initializer=\"zeros\",\n",
    "    kernel_regularizer=None,\n",
    "    bias_regularizer=None,\n",
    "    activity_regularizer=None,\n",
    "    kernel_constraint=None,\n",
    "    bias_constraint=None,\n",
    "    **kwargs)\n",
    "\n",
    "#### Block-1 layers in the order of occurrence are as follows :\n",
    "\n",
    "- Conv2D layer- This layer creates a convolutional layer for the network. Here i am creating a layer with 32 filters and a filter size of (3,3) with padding='same' to pad the image and using the  kernel initializer he_normal. I have added 2 convolutional layers each followed by an activation and batch normalization layers.\n",
    "- Activation layer - I am using a elu activation.\n",
    "- BatchNormalization - Normalize the activations of the previous layer at each batch, i.e. applies a transformation that maintains the mean activation close to 0 and the activation standard deviation close to 1.\n",
    "- MaxPooling2D layer - Downsamples the input representation by taking the maximum value over the window defined by pool_size for each dimension along the features axis.Here i have used the pool_size as (2,2).\n",
    "- Dropout: Dropout is a technique where randomly selected neurons are ignored during training. Here i am using dropout as 0.5 which means that it will ignore half of the neurons.\n",
    "\n",
    "#### Block-2 layers in the order of occurrence are as follows :\n",
    "\n",
    "- Same layers as block-1 but the convolutional layers have 64 filters.\n",
    "\n",
    "#### Block-3 layers in the order of occurrence are as follows :\n",
    "\n",
    "- Same layers as block-1 but the convolutional layers have 128 filters.\n",
    "\n",
    "#### Block-4 layers in the order of occurrence are as follows :\n",
    "\n",
    "- Same layers as block-1 but the convolutional layers have 256 filters.\n",
    "\n",
    "#### Block-5 layers in the order of occurrence are as follows :\n",
    "\n",
    "- Flatten layer - To flatten the output of the previous layers in a falat layer or in other words in the form of a vector.\n",
    "- Dense layer - A densely connected layer where each neuron is connected to every other neuron. Here i am using 64 units or 64 neurons with a kernal initializer - he_normal.\n",
    "- These layers are followed by activation layer with elu activation , batch normalization and finally a dropout with 50% dropout.\n",
    "\n",
    "#### Block-6 layers in the order of occurrence are as follows :\n",
    "\n",
    "- Same layers as blcok 5 but without flatten layer as the input for this block is already flattened.\n",
    "\n",
    "#### Block-7 layers in the order of occurrence are as follows :\n",
    "\n",
    "- Dense layer - Finally in the final block of the network i am using num_classes to create a dense layer having units=number of classes with a he_normal initializer.\n",
    "\n",
    "- Activation layer - Here i am using a softmax layer which is used for multi-class classifications."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Too many layers, but finally it's over!!\n",
    "\n",
    "Now to check the overall structure of the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d (Conv2D)              (None, 48, 48, 32)        320       \n",
      "_________________________________________________________________\n",
      "activation (Activation)      (None, 48, 48, 32)        0         \n",
      "_________________________________________________________________\n",
      "batch_normalization (BatchNo (None, 48, 48, 32)        128       \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 48, 48, 32)        9248      \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 48, 48, 32)        0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_1 (Batch (None, 48, 48, 32)        128       \n",
      "_________________________________________________________________\n",
      "max_pooling2d (MaxPooling2D) (None, 24, 24, 32)        0         \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 24, 24, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 24, 24, 64)        18496     \n",
      "_________________________________________________________________\n",
      "activation_2 (Activation)    (None, 24, 24, 64)        0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_2 (Batch (None, 24, 24, 64)        256       \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 24, 24, 64)        36928     \n",
      "_________________________________________________________________\n",
      "activation_3 (Activation)    (None, 24, 24, 64)        0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_3 (Batch (None, 24, 24, 64)        256       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 12, 12, 64)        0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 12, 12, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 12, 12, 128)       73856     \n",
      "_________________________________________________________________\n",
      "activation_4 (Activation)    (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_4 (Batch (None, 12, 12, 128)       512       \n",
      "_________________________________________________________________\n",
      "conv2d_5 (Conv2D)            (None, 12, 12, 128)       147584    \n",
      "_________________________________________________________________\n",
      "activation_5 (Activation)    (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_5 (Batch (None, 12, 12, 128)       512       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 6, 6, 128)         0         \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 6, 6, 128)         0         \n",
      "_________________________________________________________________\n",
      "conv2d_6 (Conv2D)            (None, 6, 6, 256)         295168    \n",
      "_________________________________________________________________\n",
      "activation_6 (Activation)    (None, 6, 6, 256)         0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_6 (Batch (None, 6, 6, 256)         1024      \n",
      "_________________________________________________________________\n",
      "conv2d_7 (Conv2D)            (None, 6, 6, 256)         590080    \n",
      "_________________________________________________________________\n",
      "activation_7 (Activation)    (None, 6, 6, 256)         0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_7 (Batch (None, 6, 6, 256)         1024      \n",
      "_________________________________________________________________\n",
      "max_pooling2d_3 (MaxPooling2 (None, 3, 3, 256)         0         \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 3, 3, 256)         0         \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 2304)              0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 64)                147520    \n",
      "_________________________________________________________________\n",
      "activation_8 (Activation)    (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_8 (Batch (None, 64)                256       \n",
      "_________________________________________________________________\n",
      "dropout_4 (Dropout)          (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 64)                4160      \n",
      "_________________________________________________________________\n",
      "activation_9 (Activation)    (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_9 (Batch (None, 64)                256       \n",
      "_________________________________________________________________\n",
      "dropout_5 (Dropout)          (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 5)                 325       \n",
      "_________________________________________________________________\n",
      "activation_10 (Activation)   (None, 5)                 0         \n",
      "=================================================================\n",
      "Total params: 1,328,037\n",
      "Trainable params: 1,325,861\n",
      "Non-trainable params: 2,176\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a big network which consits of 1,328,037 parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 5:\n",
    "\n",
    "The final task!!\n",
    "\n",
    "Now the only thing left is to compile and train the model. But first let's import some more things."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.optimizers import RMSprop,SGD,Adam\n",
    "from keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's do the magic.\n",
    "\n",
    "Before compiling i will create 3 things using **keras.callbacks** class:\n",
    "\n",
    "#### Checkpoint( Function - ModelCheckpoint() )\n",
    "\n",
    "It will monitor the validation loss and will try to minimise the loss using the mode='min' property. When the checkpoint is reached it will save the best trained weights. Verbose=1 is just for visulaisation when the code created checkpoint.Here i am using it's following parameters:\n",
    "\n",
    "- filepath: Path to save the model file.Here i am saving the model file with the name EmotionDetectionModel.h5\n",
    "- monitor: Quantity to monitor.Here i am monitoring the validation loss.\n",
    "- mode: One of {auto, min, max}. If save_best_only=True, the decision to overwrite the current save file is made based on either the maximization or the minimization of the monitored quantity.\n",
    "- save_best_only: If save_best_only=True, the latest best model according to the quantity monitored will not be overwritten.\n",
    "- verbose: int. 0: quiet, 1: update messages.\n",
    "\n",
    "#### Early Stopping ( Function - EarlyStopping() )\n",
    "\n",
    "This will stop the execution early by checking the following properties.\n",
    "\n",
    "- monitor:  Quantity to monitor.Here i am monitoring the validation loss.\n",
    "- min_delta: Minimum change in the monitored quantity to qualify as an improvement, i.e. an absolute change of less than min_delta, will count as no improvement.Here i have given it 0.\n",
    "- patience: Number of epochs with no improvement after which training will be stopped. Here i have given it 3.\n",
    "- restore_best_weights: Whether to restore model weights from the epoch with the best value of the monitored quantity. If False, the model weights obtained at the last step of training are used.Here i have given it True.\n",
    "- verbose: int. 0: quiet, 1: update messages.\n",
    "\n",
    "#### Reduce Learning Rate ( Function - ReduceLROnPlateau() )\n",
    "\n",
    "Models often benefit from reducing the learning rate by a factor of 2-10 once learning stagnates. This callback monitors a quantity and if no improvement is seen for a 'patience' number of epochs, the learning rate is reduced. I have used the following properties for this.\n",
    "\n",
    "- monitor: To monitor a particular loss. Here i am monitoring the validation loss.\n",
    "- factor: Factor by which the learning rate will be reduced. **new_lr = lr * factor**. Here i am using 0.2 as factor.\n",
    "- patience: Number of epochs with no improvement after which learning rate will be reduced.Here i am using 3.\n",
    "- min_delta: Threshold for measuring the new optimum, to only focus on significant changes.\n",
    "- verbose: int. 0: quiet, 1: update messages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "checkpoint = ModelCheckpoint('EmotionDetectionModel.h5',\n",
    "                             monitor='val_loss',\n",
    "                             mode='min',\n",
    "                             save_best_only=True,\n",
    "                             verbose=1)\n",
    "\n",
    "earlystop = EarlyStopping(monitor='val_loss',\n",
    "                          min_delta=0,\n",
    "                          patience=3,\n",
    "                          verbose=1,\n",
    "                          restore_best_weights=True\n",
    "                          )\n",
    "\n",
    "reduce_lr = ReduceLROnPlateau(monitor='val_loss',\n",
    "                              factor=0.2,\n",
    "                              patience=3,\n",
    "                              verbose=1,\n",
    "                              min_delta=0.0001)\n",
    "\n",
    "callbacks = [earlystop,checkpoint,reduce_lr]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it's time to finally compile the model using **model.compile()** and fit or train the model on the dataset using **model.fit_generator()**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### model.compile()\n",
    "\n",
    "It has the following arguments:\n",
    "\n",
    "- loss: This value will determine the type of loss function to use in your code. Here i have categorical data in 5 categories or classes so i have used 'categorical_crossentropy' loss.\n",
    "- optimizer: This value will determine the type of optimizer function to use in your code.Here i have used Adam optimizer with learning rate 0.001 as it is the best optimizer for categorical data.\n",
    "- metrics: The metrics argument should be a list - you model can have any number of metrics.It is the list of metrics to be evaluated by the model during training and testing.Here i have used accuracy as metric which will compile mu model according to the accuracy.\n",
    "\n",
    "#### model.fit_generator()\n",
    "\n",
    "Fits the model on data yielded batch-by-batch by a Python generator.\n",
    "\n",
    "It has the following arguments:\n",
    "\n",
    "- generator: The train_generator object that we created earlier.\n",
    "- steps_per_epochs: The steps to take on the training data in one epoch.\n",
    "- epochs: The total number of epochs (pass though the whole dataset once).\n",
    "- callbacks: The list containing all the callbacks that we created earlier.\n",
    "- validation_data: The validation_generator object that we created earlier.\n",
    "- validation_steps: The steps to take on the validation data in one epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/vitaliystepanov/opt/anaconda3/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1844: UserWarning: `Model.fit_generator` is deprecated and will be removed in a future version. Please use `Model.fit`, which supports generators.\n",
      "  warnings.warn('`Model.fit_generator` is deprecated and '\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "75/75 [==============================] - 40s 505ms/step - loss: 2.5615 - accuracy: 0.2113 - val_loss: 3.2233 - val_accuracy: 0.1840\n",
      "\n",
      "Epoch 00001: val_loss improved from inf to 3.22326, saving model to EmotionDetectionModel.h5\n",
      "Epoch 2/25\n",
      "75/75 [==============================] - 36s 486ms/step - loss: 2.2117 - accuracy: 0.2120 - val_loss: 1.6960 - val_accuracy: 0.2361\n",
      "\n",
      "Epoch 00002: val_loss improved from 3.22326 to 1.69602, saving model to EmotionDetectionModel.h5\n",
      "Epoch 3/25\n",
      "75/75 [==============================] - 41s 539ms/step - loss: 2.0174 - accuracy: 0.2111 - val_loss: 1.6325 - val_accuracy: 0.2604\n",
      "\n",
      "Epoch 00003: val_loss improved from 1.69602 to 1.63249, saving model to EmotionDetectionModel.h5\n",
      "Epoch 4/25\n",
      "75/75 [==============================] - 38s 506ms/step - loss: 1.8825 - accuracy: 0.2427 - val_loss: 1.6198 - val_accuracy: 0.2292\n",
      "\n",
      "Epoch 00004: val_loss improved from 1.63249 to 1.61975, saving model to EmotionDetectionModel.h5\n",
      "Epoch 5/25\n",
      "75/75 [==============================] - 37s 492ms/step - loss: 1.8683 - accuracy: 0.2190 - val_loss: 1.5884 - val_accuracy: 0.2674\n",
      "\n",
      "Epoch 00005: val_loss improved from 1.61975 to 1.58836, saving model to EmotionDetectionModel.h5\n",
      "Epoch 6/25\n",
      "75/75 [==============================] - 36s 476ms/step - loss: 1.7803 - accuracy: 0.2359 - val_loss: 1.6238 - val_accuracy: 0.2778\n",
      "\n",
      "Epoch 00006: val_loss did not improve from 1.58836\n",
      "Epoch 7/25\n",
      "75/75 [==============================] - 37s 494ms/step - loss: 1.7242 - accuracy: 0.2394 - val_loss: 1.5349 - val_accuracy: 0.3056\n",
      "\n",
      "Epoch 00007: val_loss improved from 1.58836 to 1.53489, saving model to EmotionDetectionModel.h5\n",
      "Epoch 8/25\n",
      "75/75 [==============================] - 35s 461ms/step - loss: 1.6932 - accuracy: 0.2407 - val_loss: 1.5724 - val_accuracy: 0.2847\n",
      "\n",
      "Epoch 00008: val_loss did not improve from 1.53489\n",
      "Epoch 9/25\n",
      "75/75 [==============================] - 34s 456ms/step - loss: 1.6640 - accuracy: 0.2476 - val_loss: 1.5381 - val_accuracy: 0.3125\n",
      "\n",
      "Epoch 00009: val_loss did not improve from 1.53489\n",
      "Epoch 10/25\n",
      "75/75 [==============================] - 34s 452ms/step - loss: 1.6086 - accuracy: 0.2662 - val_loss: 1.5176 - val_accuracy: 0.3264\n",
      "\n",
      "Epoch 00010: val_loss improved from 1.53489 to 1.51761, saving model to EmotionDetectionModel.h5\n",
      "Epoch 11/25\n",
      "75/75 [==============================] - 34s 454ms/step - loss: 1.6168 - accuracy: 0.2456 - val_loss: 1.5494 - val_accuracy: 0.2604\n",
      "\n",
      "Epoch 00011: val_loss did not improve from 1.51761\n",
      "Epoch 12/25\n",
      "75/75 [==============================] - 36s 475ms/step - loss: 1.6104 - accuracy: 0.2614 - val_loss: 1.5630 - val_accuracy: 0.2708\n",
      "\n",
      "Epoch 00012: val_loss did not improve from 1.51761\n",
      "Epoch 13/25\n",
      "75/75 [==============================] - 41s 551ms/step - loss: 1.6077 - accuracy: 0.2648 - val_loss: 1.5494 - val_accuracy: 0.2778\n",
      "Restoring model weights from the end of the best epoch.\n",
      "\n",
      "Epoch 00013: val_loss did not improve from 1.51761\n",
      "\n",
      "Epoch 00013: ReduceLROnPlateau reducing learning rate to 0.00020000000949949026.\n",
      "Epoch 00013: early stopping\n"
     ]
    }
   ],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer = Adam(lr=0.001),\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "nb_train_samples = 2417\n",
    "nb_validation_samples = 300\n",
    "epochs=25\n",
    "\n",
    "history=model.fit_generator(\n",
    "                train_generator,\n",
    "                steps_per_epoch=nb_train_samples//batch_size,\n",
    "                epochs=epochs,\n",
    "                callbacks=callbacks,\n",
    "                validation_data=validation_generator,\n",
    "                validation_steps=nb_validation_samples//batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DONE!!\n",
    "\n",
    "The model generation is completed now you can use this model to create the emotion detector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}