Gesture_Recognition.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gesture Recognition\n",
    "In this group project, you are going to build a 3D Conv model that will be able to predict the 5 gestures correctly. Please import the following libraries to get started. Once you have completed the code you can download the notebook for making a submission."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import os\n",
    "from imageio import imread\n",
    "from skimage.transform import resize\n",
    "import datetime\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We set the random seed so that the results don't vary drastically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "np.random.seed(30)\n",
    "import random as rn\n",
    "rn.seed(30)\n",
    "from tensorflow import keras\n",
    "import tensorflow as tf\n",
    "tf.random.set_seed(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this block, you read the folder names for training and validation. You also set the `batch_size` here. Note that you set the batch size in such a way that you are able to use the GPU in full capacity. You keep increasing the batch size until the machine throws an error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**data path: /home/datasets/Project_data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_doc = np.random.permutation(open('/home/datasets/Project_data/train.csv').readlines())\n",
    "val_doc = np.random.permutation(open('/home/datasets/Project_data/val.csv').readlines())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# no of images to be considered per video\n",
    "x = 30\n",
    "\n",
    "# Dimensions of image\n",
    "y=120\n",
    "z=120"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generator\n",
    "This is one of the most important part of the code. The overall structure of the generator has been given. In the generator, you are going to preprocess the images as you have images of 2 different dimensions as well as create a batch of video frames. You have to experiment with `img_idx`, `y`,`z` and normalization such that you get high accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generator(source_path, folder_list, batch_size):\n",
    " \n",
    "    print('Source path = ', source_path, '; batch size =', batch_size)\n",
    "\n",
    "    # Sampling x evenly spaced frames from each video\n",
    "    img_idx = np.linspace(0, 29, x).astype(int)  \n",
    "\n",
    "    def process_batch(start_idx, end_idx):\n",
    "        current_batch_size = end_idx - start_idx\n",
    "        batch_data = np.zeros((current_batch_size, x, y, z, 3))  # Preallocate for RGB images\n",
    "        batch_labels = np.zeros((current_batch_size, 5))  # Assuming 5 classes for one-hot encoding\n",
    "\n",
    "        for folder_idx in range(current_batch_size):\n",
    "            folder_path = source_path + '/' + folder_list[start_idx + folder_idx].split(';')[0]\n",
    "            label = int(folder_list[start_idx + folder_idx].strip().split(';')[2])\n",
    "\n",
    "            imgs = sorted(os.listdir(folder_path))  # Sort to ensure frame order consistency\n",
    "\n",
    "            for idx, item in enumerate(img_idx):\n",
    "                image_path = folder_path + '/' + imgs[item]\n",
    "                image = imread(image_path).astype(np.float32)\n",
    "                \n",
    "                # Cropping (example: crop the center region)\n",
    "                crop_size = min(image.shape[:2])  # Get the smaller dimension (assuming the crop is square)\n",
    "                center = (image.shape[0] // 2, image.shape[1] // 2)  # Find the center of the image\n",
    "                crop_image = image[\n",
    "                    center[0] - crop_size // 2 : center[0] + crop_size // 2, \n",
    "                    center[1] - crop_size // 2 : center[1] + crop_size // 2\n",
    "                ]\n",
    "\n",
    "                    \n",
    "                # Resize the image to (y, z)\n",
    "                image_resized = resize(crop_image, (y, z))\n",
    "\n",
    "                # Normalizing and Assigning each channel explicitly\n",
    "                batch_data[folder_idx, idx, :, :, 0] = image_resized[:, :, 0] / 255.0  # Normalize R channel\n",
    "                batch_data[folder_idx, idx, :, :, 1] = image_resized[:, :, 1] / 255.0  # Normalize G channel\n",
    "                batch_data[folder_idx, idx, :, :, 2] = image_resized[:, :, 2] / 255.0  # Normalize B channel\n",
    "\n",
    "            batch_labels[folder_idx, label] = 1  # One-hot encode the label\n",
    "\n",
    "        return batch_data, batch_labels\n",
    "\n",
    "    while True:\n",
    "        t = np.random.permutation(folder_list)  # Shuffle the folder list\n",
    "        num_batches = len(folder_list) // batch_size  # Calculate the number of full batches\n",
    "\n",
    "        for batch in range(num_batches):\n",
    "            start_idx = batch * batch_size\n",
    "            end_idx = start_idx + batch_size\n",
    "            yield process_batch(start_idx, end_idx)\n",
    "\n",
    "        # Handle the remaining data points after full batches\n",
    "        remaining_folders = len(folder_list) % batch_size\n",
    "        if remaining_folders > 0:\n",
    "            start_idx = num_batches * batch_size\n",
    "            end_idx = len(folder_list)\n",
    "            yield process_batch(start_idx, end_idx)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note here that a video is represented above in the generator as (number of images, height, width, number of channels). Take this into consideration while creating the model architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# training sequences = 663\n",
      "# validation sequences = 100\n"
     ]
    }
   ],
   "source": [
    "curr_dt_time = datetime.datetime.now()\n",
    "train_path = '/home/datasets/Project_data/train'\n",
    "val_path = '/home/datasets/Project_data/val'\n",
    "num_train_sequences = len(train_doc)\n",
    "print('# training sequences =', num_train_sequences)\n",
    "num_val_sequences = len(val_doc)\n",
    "print('# validation sequences =', num_val_sequences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 30\n",
    "num_epochs = 5\n",
    "train_generator = generator(train_path, train_doc, batch_size)\n",
    "val_generator = generator(val_path, val_doc, batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "if (num_train_sequences%batch_size) == 0:\n",
    "    steps_per_epoch = int(num_train_sequences/batch_size)\n",
    "else:\n",
    "    steps_per_epoch = (num_train_sequences//batch_size) + 1\n",
    "\n",
    "if (num_val_sequences%batch_size) == 0:\n",
    "    validation_steps = int(num_val_sequences/batch_size)\n",
    "else:\n",
    "    validation_steps = (num_val_sequences//batch_size) + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "77"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# clearing the cache\n",
    "import gc\n",
    "tf.keras.backend.clear_session()\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# All imports\n",
    "from keras.models import Sequential, Model\n",
    "from keras.layers import Dense, GRU, Flatten, LSTM,GRU,TimeDistributed, Dense,Dropout,Input, BatchNormalization, Activation, GlobalAveragePooling3D,GlobalAveragePooling2D,Bidirectional\n",
    "from keras.layers.convolutional import Conv3D, MaxPooling3D\n",
    "from keras.callbacks import ModelCheckpoint, ReduceLROnPlateau,EarlyStopping\n",
    "\n",
    "from tensorflow.keras.regularizers import l2\n",
    "from tensorflow.keras.applications import ResNet50\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "from keras import optimizers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model\n",
    "Here you make the model using different functionalities that Keras provides. Remember to use `Conv3D` and `MaxPooling3D` and not `Conv2D` and `Maxpooling2D` for a 3D convolution model. You would want to use `TimeDistributed` while building a Conv2D + RNN model. Also remember that the last layer is the softmax. Design the network in such a way that the model is able to give good accuracy on the least number of parameters so that it can fit in the memory of the webcam."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_conv3d_model(input_shape, num_classes):\n",
    "\n",
    "    model = Sequential()\n",
    "\n",
    "    # Conv3D Block 1\n",
    "    model.add(Conv3D(64, kernel_size=(3, 3, 3), activation='relu', padding='same', kernel_regularizer=l2(0.01),input_shape=input_shape))\n",
    "    model.add(BatchNormalization())\n",
    "    model.add(MaxPooling3D(pool_size=(2, 2, 2)))\n",
    "    model.add(Dropout(0.5))\n",
    "\n",
    "    # Conv3D Block 2\n",
    "    model.add(Conv3D(128, kernel_size=(3, 3, 3), activation='relu',kernel_regularizer=l2(0.01), padding='same'))\n",
    "    model.add(BatchNormalization())\n",
    "    model.add(MaxPooling3D(pool_size=(2, 2, 2)))\n",
    "    model.add(Dropout(0.5))\n",
    "\n",
    "    # Conv3D Block 3\n",
    "    model.add(Conv3D(256, kernel_size=(3, 3, 3), activation='relu',kernel_regularizer=l2(0.01), padding='same'))\n",
    "    model.add(BatchNormalization())\n",
    "    model.add(MaxPooling3D(pool_size=(2, 2, 2)))\n",
    "    model.add(Dropout(0.5))\n",
    "\n",
    "    # Global Average Pooling (instead of Flatten)\n",
    "    model.add(GlobalAveragePooling3D())\n",
    "    \n",
    "    # Fully Connected Layer\n",
    "    model.add(Dense(512, activation='relu'))\n",
    "    model.add(Dropout(0.7))\n",
    "\n",
    "    # Output Layer\n",
    "    model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "    return model\n",
    "\n",
    "input_shape = (x,y,z, 3)  # (frames, height, width, channels)\n",
    "num_classes = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have written the model, the next step is to `compile` the model. When you print the `summary` of the model, you'll see the total number of parameters you have to train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " conv3d_3 (Conv3D)           (None, 30, 120, 120, 64)  5248      \n",
      "                                                                 \n",
      " batch_normalization_3 (Batc  (None, 30, 120, 120, 64)  256      \n",
      " hNormalization)                                                 \n",
      "                                                                 \n",
      " max_pooling3d_3 (MaxPooling  (None, 15, 60, 60, 64)   0         \n",
      " 3D)                                                             \n",
      "                                                                 \n",
      " dropout_4 (Dropout)         (None, 15, 60, 60, 64)    0         \n",
      "                                                                 \n",
      " conv3d_4 (Conv3D)           (None, 15, 60, 60, 128)   221312    \n",
      "                                                                 \n",
      " batch_normalization_4 (Batc  (None, 15, 60, 60, 128)  512       \n",
      " hNormalization)                                                 \n",
      "                                                                 \n",
      " max_pooling3d_4 (MaxPooling  (None, 7, 30, 30, 128)   0         \n",
      " 3D)                                                             \n",
      "                                                                 \n",
      " dropout_5 (Dropout)         (None, 7, 30, 30, 128)    0         \n",
      "                                                                 \n",
      " conv3d_5 (Conv3D)           (None, 7, 30, 30, 256)    884992    \n",
      "                                                                 \n",
      " batch_normalization_5 (Batc  (None, 7, 30, 30, 256)   1024      \n",
      " hNormalization)                                                 \n",
      "                                                                 \n",
      " max_pooling3d_5 (MaxPooling  (None, 3, 15, 15, 256)   0         \n",
      " 3D)                                                             \n",
      "                                                                 \n",
      " dropout_6 (Dropout)         (None, 3, 15, 15, 256)    0         \n",
      "                                                                 \n",
      " global_average_pooling3d_1   (None, 256)              0         \n",
      " (GlobalAveragePooling3D)                                        \n",
      "                                                                 \n",
      " dense_2 (Dense)             (None, 512)               131584    \n",
      "                                                                 \n",
      " dropout_7 (Dropout)         (None, 512)               0         \n",
      "                                                                 \n",
      " dense_3 (Dense)             (None, 5)                 2565      \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 1,247,493\n",
      "Trainable params: 1,246,597\n",
      "Non-trainable params: 896\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "model = create_conv3d_model(input_shape, num_classes)\n",
    "\n",
    "optimizer = Adam(learning_rate=0.001)  # You can adjust the learning rate as needed\n",
    "model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "print (model.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# callbacks\n",
    "model_name = 'Conv3D' + '_' + str(curr_dt_time).replace(' ','').replace(':','_') + '/'\n",
    "    \n",
    "if not os.path.exists(model_name):\n",
    "    os.mkdir(model_name)\n",
    "        \n",
    "filepath = model_name + 'model-{epoch:05d}-{loss:.5f}-{categorical_accuracy:.5f}-{val_loss:.5f}-{val_categorical_accuracy:.5f}.h5'\n",
    "\n",
    "checkpoint = ModelCheckpoint(filepath, monitor='val_loss', verbose=1, save_best_only=True, save_weights_only=True, mode='auto',save_freq = 'epoch')\n",
    "\n",
    "LR = ReduceLROnPlateau(\n",
    "    monitor='val_loss',       # Metric to monitor (e.g., validation loss)\n",
    "    factor=0.5,               # Factor by which the learning rate will be reduced\n",
    "    patience=3,               # Number of epochs with no improvement before reducing the learning rate\n",
    "    min_lr=1e-6,              # Lower bound on the learning rate\n",
    "    verbose=1                 # Print updates when learning rate changes\n",
    ")\n",
    "early_stopping = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)\n",
    "\n",
    "callbacks_list = [early_stopping,checkpoint, LR]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `steps_per_epoch` and `validation_steps` are used by `fit` method to decide the number of next() calls it need to make."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now fit the model. This will start training the model and with the help of the checkpoints, you'll be able to save the model at the end of each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source path =  /home/datasets/Project_data/train ; batch size = 10\n",
      "Epoch 1/5\n",
      "67/67 [==============================] - ETA: 0s - loss: 3.2749 - categorical_accuracy: 0.3891Source path =  /home/datasets/Project_data/val ; batch size = 10\n",
      "\n",
      "Epoch 00001: saving model to Conv3D_2024-12-0921_30_00.698437/model-00001-3.27491-0.38914-3.44845-0.21000.h5\n",
      "67/67 [==============================] - 175s 3s/step - loss: 3.2749 - categorical_accuracy: 0.3891 - val_loss: 3.4485 - val_categorical_accuracy: 0.2100 - lr: 0.0010\n",
      "Epoch 2/5\n",
      "67/67 [==============================] - ETA: 0s - loss: 2.0318 - categorical_accuracy: 0.4329\n",
      "Epoch 00002: saving model to Conv3D_2024-12-0921_30_00.698437/model-00002-2.03182-0.43288-3.11014-0.22000.h5\n",
      "67/67 [==============================] - 159s 2s/step - loss: 2.0318 - categorical_accuracy: 0.4329 - val_loss: 3.1101 - val_categorical_accuracy: 0.2200 - lr: 0.0010\n",
      "Epoch 3/5\n",
      "67/67 [==============================] - ETA: 0s - loss: 1.6520 - categorical_accuracy: 0.4917\n",
      "Epoch 00003: saving model to Conv3D_2024-12-0921_30_00.698437/model-00003-1.65203-0.49170-2.29201-0.29000.h5\n",
      "67/67 [==============================] - 148s 2s/step - loss: 1.6520 - categorical_accuracy: 0.4917 - val_loss: 2.2920 - val_categorical_accuracy: 0.2900 - lr: 0.0010\n",
      "Epoch 4/5\n",
      "67/67 [==============================] - ETA: 0s - loss: 1.5121 - categorical_accuracy: 0.5173\n",
      "Epoch 00004: saving model to Conv3D_2024-12-0921_30_00.698437/model-00004-1.51210-0.51735-3.23031-0.16000.h5\n",
      "67/67 [==============================] - 150s 2s/step - loss: 1.5121 - categorical_accuracy: 0.5173 - val_loss: 3.2303 - val_categorical_accuracy: 0.1600 - lr: 0.0010\n",
      "Epoch 5/5\n",
      "67/67 [==============================] - ETA: 0s - loss: 1.4031 - categorical_accuracy: 0.5551\n",
      "Epoch 00005: saving model to Conv3D_2024-12-0921_30_00.698437/model-00005-1.40309-0.55505-4.66842-0.16000.h5\n",
      "67/67 [==============================] - 147s 2s/step - loss: 1.4031 - categorical_accuracy: 0.5551 - val_loss: 4.6684 - val_categorical_accuracy: 0.1600 - lr: 0.0010\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f8a616e7850>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating 2 seperate CNN and RNN layers with transfer learning from Resnet50\n",
    "# Freezing Resnet and using simple RNN\n",
    "\n",
    "def create_transfer_cnn_lstm_model(input_shape, num_classes):\n",
    "    # Input shape is (number_of_frames, height, width, channels)\n",
    "    inputs = Input(shape=input_shape)\n",
    "\n",
    "    # Pre-trained ResNet50 model\n",
    "    base_model = ResNet50(\n",
    "        weights=\"imagenet\", include_top=False, input_shape=(input_shape[1], input_shape[2], input_shape[3])\n",
    "    )\n",
    "    base_model.trainable = False  # Freeze the pre-trained weights\n",
    "\n",
    "    # CNN Module using TimeDistributed\n",
    "    cnn_model = Sequential(name=\"CNN_Module\")\n",
    "    cnn_model.add(base_model)\n",
    "    cnn_model.add(GlobalAveragePooling2D())\n",
    "\n",
    "    time_distributed_cnn = TimeDistributed(cnn_model, name=\"TimeDistributed_CNN\")(inputs)\n",
    "\n",
    "    # RNN Module\n",
    "    lstm_layer = LSTM(128, return_sequences=False, dropout=0.5, recurrent_dropout=0.3, name=\"LSTM_Layer\")(\n",
    "        time_distributed_cnn\n",
    "    )\n",
    "    outputs = Dense(num_classes, activation=\"softmax\", name=\"Output_Layer\")(lstm_layer)\n",
    "\n",
    "    # Final Model\n",
    "    model = Model(inputs=inputs, outputs=outputs, name=\"ResNet50_CNN_RNN_Model\")\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer LSTM_Layer will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "Model: \"ResNet50_CNN_RNN_Model\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_1 (InputLayer)        [(None, 30, 120, 120, 3)  0         \n",
      "                             ]                                   \n",
      "                                                                 \n",
      " TimeDistributed_CNN (TimeDi  (None, 30, 2048)         23587712  \n",
      " stributed)                                                      \n",
      "                                                                 \n",
      " LSTM_Layer (LSTM)           (None, 128)               1114624   \n",
      "                                                                 \n",
      " Output_Layer (Dense)        (None, 5)                 645       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 24,702,981\n",
      "Trainable params: 1,115,269\n",
      "Non-trainable params: 23,587,712\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "model2 = create_transfer_cnn_lstm_model(input_shape, num_classes)\n",
    "optimizer = Adam(learning_rate=0.001)  # You can adjust the learning rate as needed\n",
    "model2.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "print (model2.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source path =  /home/datasets/Project_data/train ; batch size = 30\n",
      "Epoch 1/5\n",
      "23/23 [==============================] - ETA: 0s - loss: 1.6186 - categorical_accuracy: 0.2443Source path =  /home/datasets/Project_data/val ; batch size = 30\n",
      "\n",
      "Epoch 00001: val_loss improved from inf to 1.56808, saving model to Conv3D_2024-12-1001_09_04.144053/model-00001-1.61855-0.24434-1.56808-0.24000.h5\n",
      "23/23 [==============================] - 155s 7s/step - loss: 1.6186 - categorical_accuracy: 0.2443 - val_loss: 1.5681 - val_categorical_accuracy: 0.2400 - lr: 0.0010\n",
      "Epoch 2/5\n",
      "23/23 [==============================] - ETA: 0s - loss: 1.5947 - categorical_accuracy: 0.2579\n",
      "Epoch 00002: val_loss improved from 1.56808 to 1.50348, saving model to Conv3D_2024-12-1001_09_04.144053/model-00002-1.59475-0.25792-1.50348-0.24000.h5\n",
      "23/23 [==============================] - 149s 7s/step - loss: 1.5947 - categorical_accuracy: 0.2579 - val_loss: 1.5035 - val_categorical_accuracy: 0.2400 - lr: 0.0010\n",
      "Epoch 3/5\n",
      "20/23 [=========================>....] - ETA: 17s - loss: 1.5138 - categorical_accuracy: 0.3117"
     ]
    }
   ],
   "source": [
    "model2.fit(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# changes to above model\n",
    "# allowing last 20 layers of Resnet to learn\n",
    "# Using BiDirenctional LSTM insted of simple LSTM\n",
    "\n",
    "def create_transfer_cnn_bi_lstm_model(input_shape, num_classes):\n",
    "    # Input shape is (number_of_frames, height, width, channels)\n",
    "    inputs = Input(shape=input_shape)\n",
    "\n",
    "    # Pre-trained ResNet50 model\n",
    "    base_model = ResNet50(\n",
    "        weights=\"imagenet\", include_top=False, input_shape=(input_shape[1], input_shape[2], input_shape[3])\n",
    "    )\n",
    "    base_model.trainable = False  # Freeze the pre-trained weights\n",
    "\n",
    "    # Unfreeze the last few layers for fine-tuning\n",
    "    for layer in base_model.layers[-20:]:\n",
    "        layer.trainable = True\n",
    "\n",
    "    # CNN Module with L2 regularization\n",
    "    cnn_model = Sequential(name=\"CNN_Module\")\n",
    "    cnn_model.add(base_model)\n",
    "    cnn_model.add(GlobalAveragePooling2D())\n",
    "    cnn_model.add(Dense(256, activation='relu', kernel_regularizer=l2(0.01)))\n",
    "    cnn_model.add(Dropout(0.5))\n",
    "\n",
    "    # TimeDistributed CNN Layer\n",
    "    time_distributed_cnn = TimeDistributed(cnn_model, name=\"TimeDistributed_CNN\")(inputs)\n",
    "\n",
    "    # RNN Module with Bidirectional LSTM and L2 regularization\n",
    "    lstm_layer = Bidirectional(LSTM(256, return_sequences=False, dropout=0.5, recurrent_dropout=0.3, kernel_regularizer=l2(0.01)), name=\"Bidirectional_LSTM_Layer\")(time_distributed_cnn)\n",
    "    outputs = Dense(num_classes, activation=\"softmax\", name=\"Output_Layer\")(lstm_layer)\n",
    "\n",
    "    # Final Model\n",
    "    model = Model(inputs=inputs, outputs=outputs, name=\"ResNet50_CNN_RNN_Model\")\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer lstm will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "WARNING:tensorflow:Layer lstm will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "WARNING:tensorflow:Layer lstm will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "Model: \"ResNet50_CNN_RNN_Model\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_3 (InputLayer)        [(None, 30, 120, 120, 3)  0         \n",
      "                             ]                                   \n",
      "                                                                 \n",
      " TimeDistributed_CNN (TimeDi  (None, 30, 256)          24112256  \n",
      " stributed)                                                      \n",
      "                                                                 \n",
      " Bidirectional_LSTM_Layer (B  (None, 512)              1050624   \n",
      " idirectional)                                                   \n",
      "                                                                 \n",
      " Output_Layer (Dense)        (None, 5)                 2565      \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 25,165,445\n",
      "Trainable params: 1,577,733\n",
      "Non-trainable params: 23,587,712\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "model3 = create_transfer_cnn_bi_lstm_model(input_shape, num_classes)\n",
    "optimizer = Adam(learning_rate=0.001)  # You can adjust the learning rate as needed\n",
    "model3.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "print (model3.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source path =  /home/datasets/Project_data/train ; batch size = 30\n",
      "Epoch 1/5\n",
      "23/23 [==============================] - ETA: 0s - loss: 10.5335 - categorical_accuracy: 0.3439Source path =  /home/datasets/Project_data/val ; batch size = 30\n",
      "23/23 [==============================] - 159s 7s/step - loss: 10.5335 - categorical_accuracy: 0.3439 - val_loss: 7.5909 - val_categorical_accuracy: 0.1800 - lr: 0.0010\n",
      "Epoch 2/5\n",
      "23/23 [==============================] - 149s 7s/step - loss: 5.3837 - categorical_accuracy: 0.4766 - val_loss: 4.6400 - val_categorical_accuracy: 0.1800 - lr: 0.0010\n",
      "Epoch 3/5\n",
      "23/23 [==============================] - 148s 7s/step - loss: 3.2338 - categorical_accuracy: 0.6139 - val_loss: 3.5458 - val_categorical_accuracy: 0.3100 - lr: 0.0010\n",
      "Epoch 4/5\n",
      "23/23 [==============================] - 143s 6s/step - loss: 2.3732 - categorical_accuracy: 0.6335 - val_loss: 3.2592 - val_categorical_accuracy: 0.2300 - lr: 0.0010\n",
      "Epoch 5/5\n",
      "23/23 [==============================] - 149s 7s/step - loss: 1.9887 - categorical_accuracy: 0.6591 - val_loss: 5.0402 - val_categorical_accuracy: 0.2300 - lr: 0.0010\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f89d6b63f10>"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model3.fit(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "# changes done\n",
    "# allowing 40 layers of resnet to learn\n",
    "# increased params for Dense layer\n",
    "def create_transfer_cnn_bi_lstm_model2(input_shape, num_classes):\n",
    "    # Input shape is (number_of_frames, height, width, channels)\n",
    "    inputs = Input(shape=input_shape)\n",
    "\n",
    "    # Pre-trained ResNet50 model\n",
    "    base_model = ResNet50(\n",
    "        weights=\"imagenet\", include_top=False, input_shape=(input_shape[1], input_shape[2], input_shape[3])\n",
    "    )\n",
    "    base_model.trainable = False  # Freeze the pre-trained weights\n",
    "\n",
    "    # Unfreeze the last few layers for fine-tuning\n",
    "    for layer in base_model.layers[-40:]:\n",
    "        layer.trainable = True\n",
    "\n",
    "    # CNN Module with L2 regularization\n",
    "    cnn_model = Sequential(name=\"CNN_Module\")\n",
    "    cnn_model.add(base_model)\n",
    "    cnn_model.add(GlobalAveragePooling2D())\n",
    "    cnn_model.add(Dense(512, activation='relu', kernel_regularizer=l2(0.01)))\n",
    "    cnn_model.add(Dropout(0.4))\n",
    "\n",
    "    # TimeDistributed CNN Layer\n",
    "    time_distributed_cnn = TimeDistributed(cnn_model, name=\"TimeDistributed_CNN\")(inputs)\n",
    "\n",
    "    # RNN Module with Bidirectional LSTM and L2 regularization\n",
    "    lstm_layer = Bidirectional(LSTM(256, return_sequences=False, dropout=0.4, recurrent_dropout=0.4, kernel_regularizer=l2(0.01)), name=\"Bidirectional_LSTM_Layer\")(time_distributed_cnn)\n",
    "    outputs = Dense(num_classes, activation=\"softmax\", name=\"Output_Layer\")(lstm_layer)\n",
    "\n",
    "    # Final Model\n",
    "    model = Model(inputs=inputs, outputs=outputs, name=\"ResNet50_CNN_RNN_Model\")\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer lstm_1 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "WARNING:tensorflow:Layer lstm_1 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "WARNING:tensorflow:Layer lstm_1 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "Model: \"ResNet50_CNN_RNN_Model\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_5 (InputLayer)        [(None, 30, 120, 120, 3)  0         \n",
      "                             ]                                   \n",
      "                                                                 \n",
      " TimeDistributed_CNN (TimeDi  (None, 30, 512)          24636800  \n",
      " stributed)                                                      \n",
      "                                                                 \n",
      " Bidirectional_LSTM_Layer (B  (None, 512)              1574912   \n",
      " idirectional)                                                   \n",
      "                                                                 \n",
      " Output_Layer (Dense)        (None, 5)                 2565      \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 26,214,277\n",
      "Trainable params: 2,626,565\n",
      "Non-trainable params: 23,587,712\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "model4 = create_transfer_cnn_bi_lstm_model2(input_shape, num_classes)\n",
    "optimizer = Adam(learning_rate=0.001)  # You can adjust the learning rate as needed\n",
    "model4.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "print (model4.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source path =  /home/datasets/Project_data/train ; batch size = 30\n",
      "Epoch 1/5\n",
      "23/23 [==============================] - ETA: 0s - loss: 17.1959 - categorical_accuracy: 0.3092Source path =  /home/datasets/Project_data/val ; batch size = 30\n",
      "23/23 [==============================] - 159s 7s/step - loss: 17.1959 - categorical_accuracy: 0.3092 - val_loss: 11.7062 - val_categorical_accuracy: 0.1600 - lr: 0.0010\n",
      "Epoch 2/5\n",
      "23/23 [==============================] - 150s 7s/step - loss: 8.2201 - categorical_accuracy: 0.4751 - val_loss: 6.3495 - val_categorical_accuracy: 0.1900 - lr: 0.0010\n",
      "Epoch 3/5\n",
      "23/23 [==============================] - 150s 7s/step - loss: 4.6493 - categorical_accuracy: 0.5445 - val_loss: 4.4078 - val_categorical_accuracy: 0.2200 - lr: 0.0010\n",
      "Epoch 4/5\n",
      "23/23 [==============================] - 146s 7s/step - loss: 3.2007 - categorical_accuracy: 0.6018 - val_loss: 3.4146 - val_categorical_accuracy: 0.2700 - lr: 0.0010\n",
      "Epoch 5/5\n",
      "23/23 [==============================] - 154s 7s/step - loss: 2.4814 - categorical_accuracy: 0.6501 - val_loss: 2.7880 - val_categorical_accuracy: 0.3300 - lr: 0.0010\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f8ad93e6250>"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model4.fit(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_transfer_cnn_bi_gru_model(input_shape, num_classes):\n",
    "    # Input layer\n",
    "    inputs = Input(shape=input_shape)\n",
    "\n",
    "    # Pre-trained ResNet50 model (transfer learning)\n",
    "    base_model = ResNet50(weights=\"imagenet\", include_top=False, input_shape=(input_shape[1], input_shape[2], input_shape[3]))\n",
    "    base_model.trainable = False\n",
    "    \n",
    "    # Unfreeze the last few layers for fine-tuning\n",
    "    for layer in base_model.layers[-20:]:\n",
    "        layer.trainable = True\n",
    "\n",
    "    # CNN Module with L2 regularization\n",
    "    cnn_model = Sequential(name=\"CNN_Module\")\n",
    "    cnn_model.add(base_model)\n",
    "    cnn_model.add(GlobalAveragePooling2D())\n",
    "    cnn_model.add(Dense(512, activation='relu', kernel_regularizer=l2(0.01)))\n",
    "    cnn_model.add(Dropout(0.4))\n",
    "\n",
    "    # TimeDistributed CNN Layer\n",
    "    time_distributed_cnn = TimeDistributed(cnn_model, name=\"TimeDistributed_CNN\")(inputs)\n",
    "\n",
    "    # First Bidirectional GRU Layer\n",
    "    gru_layer_1 = Bidirectional(GRU(256, return_sequences=True, dropout=0.4, recurrent_dropout=0.4, kernel_regularizer=l2(0.01)), name=\"Bidirectional_GRU_Layer_1\")(time_distributed_cnn)\n",
    "\n",
    "    # Second Bidirectional GRU Layer\n",
    "    gru_layer_2 = Bidirectional(GRU(128, return_sequences=False, dropout=0.4, recurrent_dropout=0.4, kernel_regularizer=l2(0.01)), name=\"Bidirectional_GRU_Layer_2\")(gru_layer_1)\n",
    "\n",
    "    # Output layer\n",
    "    outputs = Dense(num_classes, activation=\"softmax\", name=\"Output_Layer\")(gru_layer_2)\n",
    "\n",
    "    # Final Model\n",
    "    model = Model(inputs=inputs, outputs=outputs, name=\"ResNet50_CNN_BiGRU_Model\")\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer gru will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "WARNING:tensorflow:Layer gru will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "WARNING:tensorflow:Layer gru will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "WARNING:tensorflow:Layer gru_1 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "WARNING:tensorflow:Layer gru_1 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "WARNING:tensorflow:Layer gru_1 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "Model: \"ResNet50_CNN_BiGRU_Model\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_7 (InputLayer)        [(None, 30, 120, 120, 3)  0         \n",
      "                             ]                                   \n",
      "                                                                 \n",
      " TimeDistributed_CNN (TimeDi  (None, 30, 512)          24636800  \n",
      " stributed)                                                      \n",
      "                                                                 \n",
      " Bidirectional_GRU_Layer_1 (  (None, 30, 512)          1182720   \n",
      " Bidirectional)                                                  \n",
      "                                                                 \n",
      " Bidirectional_GRU_Layer_2 (  (None, 256)              493056    \n",
      " Bidirectional)                                                  \n",
      "                                                                 \n",
      " Output_Layer (Dense)        (None, 5)                 1285      \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 26,313,861\n",
      "Trainable params: 2,726,149\n",
      "Non-trainable params: 23,587,712\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "model5 = create_transfer_cnn_bi_gru_model(input_shape, num_classes)\n",
    "optimizer = Adam(learning_rate=0.001)  # You can adjust the learning rate as needed\n",
    "model5.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "print (model5.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source path =  /home/datasets/Project_data/train ; batch size = 30\n",
      "Epoch 1/5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-12-10 00:16:56.425665: I tensorflow/stream_executor/cuda/cuda_dnn.cc:377] Loaded cuDNN version 8302\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "23/23 [==============================] - ETA: 0s - loss: 22.9844 - categorical_accuracy: 0.3107Source path =  /home/datasets/Project_data/val ; batch size = 30\n",
      "\n",
      "Epoch 00001: val_loss improved from inf to 15.65347, saving model to Conv3D_2024-12-1000_15_50.721424/model-00001-22.98441-0.31071-15.65347-0.34000.h5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/keras/engine/functional.py:1410: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.\n",
      "  layer_config = serialize_layer_fn(layer)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "23/23 [==============================] - 165s 7s/step - loss: 22.9844 - categorical_accuracy: 0.3107 - val_loss: 15.6535 - val_categorical_accuracy: 0.3400 - lr: 0.0010\n",
      "Epoch 2/5\n",
      "23/23 [==============================] - ETA: 0s - loss: 11.5936 - categorical_accuracy: 0.4751\n",
      "Epoch 00002: val_loss improved from 15.65347 to 8.91108, saving model to Conv3D_2024-12-1000_15_50.721424/model-00002-11.59361-0.47511-8.91108-0.18000.h5\n",
      "23/23 [==============================] - 153s 7s/step - loss: 11.5936 - categorical_accuracy: 0.4751 - val_loss: 8.9111 - val_categorical_accuracy: 0.1800 - lr: 0.0010\n",
      "Epoch 3/5\n",
      "23/23 [==============================] - ETA: 0s - loss: 6.6962 - categorical_accuracy: 0.6063\n",
      "Epoch 00003: val_loss improved from 8.91108 to 5.94155, saving model to Conv3D_2024-12-1000_15_50.721424/model-00003-6.69616-0.60633-5.94155-0.26000.h5\n",
      "23/23 [==============================] - 151s 7s/step - loss: 6.6962 - categorical_accuracy: 0.6063 - val_loss: 5.9416 - val_categorical_accuracy: 0.2600 - lr: 0.0010\n",
      "Epoch 4/5\n",
      "23/23 [==============================] - ETA: 0s - loss: 4.6089 - categorical_accuracy: 0.6576\n",
      "Epoch 00004: val_loss improved from 5.94155 to 4.85647, saving model to Conv3D_2024-12-1000_15_50.721424/model-00004-4.60891-0.65762-4.85647-0.23000.h5\n",
      "23/23 [==============================] - 147s 7s/step - loss: 4.6089 - categorical_accuracy: 0.6576 - val_loss: 4.8565 - val_categorical_accuracy: 0.2300 - lr: 0.0010\n",
      "Epoch 5/5\n",
      "23/23 [==============================] - ETA: 0s - loss: 3.5103 - categorical_accuracy: 0.7149\n",
      "Epoch 00005: val_loss improved from 4.85647 to 4.09499, saving model to Conv3D_2024-12-1000_15_50.721424/model-00005-3.51030-0.71493-4.09499-0.23000.h5\n",
      "23/23 [==============================] - 151s 7s/step - loss: 3.5103 - categorical_accuracy: 0.7149 - val_loss: 4.0950 - val_categorical_accuracy: 0.2300 - lr: 0.0010\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fbc0c2a2760>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model5.fit(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_transfer_cnn_bi_gru_model_final(input_shape, num_classes):\n",
    "    # Input layer\n",
    "    inputs = Input(shape=input_shape)\n",
    "\n",
    "    # Pre-trained ResNet50 model (transfer learning)\n",
    "    base_model = ResNet50(weights=\"imagenet\", include_top=False, input_shape=(input_shape[1], input_shape[2], input_shape[3]))\n",
    "    base_model.trainable = False\n",
    "    \n",
    "    # Unfreeze the last few layers for fine-tuning\n",
    "    for layer in base_model.layers[-30:]:\n",
    "        layer.trainable = True\n",
    "\n",
    "    # CNN Module with L2 regularization\n",
    "    cnn_model = Sequential(name=\"CNN_Module\")\n",
    "    cnn_model.add(base_model)\n",
    "    cnn_model.add(GlobalAveragePooling2D())\n",
    "    cnn_model.add(Dense(512, activation='relu', kernel_regularizer=l2(0.05)))\n",
    "    cnn_model.add(Dropout(0.5))\n",
    "\n",
    "    # TimeDistributed CNN Layer\n",
    "    time_distributed_cnn = TimeDistributed(cnn_model, name=\"TimeDistributed_CNN\")(inputs)\n",
    "\n",
    "    # First Bidirectional GRU Layer\n",
    "    gru_layer_1 = Bidirectional(GRU(512, return_sequences=True, dropout=0.5, recurrent_dropout=0.5, kernel_regularizer=l2(0.05)), name=\"Bidirectional_GRU_Layer_1\")(time_distributed_cnn)\n",
    "\n",
    "    # Second Bidirectional GRU Layer\n",
    "    gru_layer_2 = Bidirectional(GRU(216, return_sequences=False, dropout=0.5, recurrent_dropout=0.5, kernel_regularizer=l2(0.05)), name=\"Bidirectional_GRU_Layer_2\")(gru_layer_1)\n",
    "\n",
    "    # Output layer\n",
    "    outputs = Dense(num_classes, activation=\"softmax\", name=\"Output_Layer\")(gru_layer_2)\n",
    "\n",
    "    # Final Model\n",
    "    model = Model(inputs=inputs, outputs=outputs, name=\"ResNet50_CNN_BiGRU_Model\")\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer gru will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "WARNING:tensorflow:Layer gru will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "WARNING:tensorflow:Layer gru will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "WARNING:tensorflow:Layer gru_1 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "WARNING:tensorflow:Layer gru_1 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "WARNING:tensorflow:Layer gru_1 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "Model: \"ResNet50_CNN_BiGRU_Model\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_1 (InputLayer)        [(None, 30, 120, 120, 3)  0         \n",
      "                             ]                                   \n",
      "                                                                 \n",
      " TimeDistributed_CNN (TimeDi  (None, 30, 512)          24636800  \n",
      " stributed)                                                      \n",
      "                                                                 \n",
      " Bidirectional_GRU_Layer_1 (  (None, 30, 1024)         3151872   \n",
      " Bidirectional)                                                  \n",
      "                                                                 \n",
      " Bidirectional_GRU_Layer_2 (  (None, 432)              1609632   \n",
      " Bidirectional)                                                  \n",
      "                                                                 \n",
      " Output_Layer (Dense)        (None, 5)                 2165      \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 29,400,469\n",
      "Trainable params: 5,812,757\n",
      "Non-trainable params: 23,587,712\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "model6 = create_transfer_cnn_bi_gru_model_final(input_shape, num_classes)\n",
    "optimizer = Adam(learning_rate=1e-4)\n",
    "model6.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "print (model6.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source path =  /home/datasets/Project_data/train ; batch size = 30\n",
      "Epoch 1/5\n",
      "23/23 [==============================] - ETA: 0s - loss: 188.8110 - categorical_accuracy: 0.2881Source path =  /home/datasets/Project_data/val ; batch size = 30\n",
      "\n",
      "Epoch 00001: val_loss improved from inf to 177.43866, saving model to Conv3D_2024-12-1002_48_49.316964/model-00001-188.81100-0.28808-177.43866-0.17000.h5\n",
      "23/23 [==============================] - 167s 7s/step - loss: 188.8110 - categorical_accuracy: 0.2881 - val_loss: 177.4387 - val_categorical_accuracy: 0.1700 - lr: 1.0000e-04\n",
      "Epoch 2/5\n",
      "23/23 [==============================] - ETA: 0s - loss: 168.2739 - categorical_accuracy: 0.3288\n",
      "Epoch 00002: val_loss improved from 177.43866 to 158.11235, saving model to Conv3D_2024-12-1002_48_49.316964/model-00002-168.27394-0.32881-158.11235-0.23000.h5\n",
      "23/23 [==============================] - 151s 7s/step - loss: 168.2739 - categorical_accuracy: 0.3288 - val_loss: 158.1124 - val_categorical_accuracy: 0.2300 - lr: 1.0000e-04\n",
      "Epoch 3/5\n",
      "23/23 [==============================] - ETA: 0s - loss: 149.7669 - categorical_accuracy: 0.4012\n",
      "Epoch 00003: val_loss improved from 158.11235 to 140.69827, saving model to Conv3D_2024-12-1002_48_49.316964/model-00003-149.76691-0.40121-140.69827-0.34000.h5\n",
      "23/23 [==============================] - 149s 7s/step - loss: 149.7669 - categorical_accuracy: 0.4012 - val_loss: 140.6983 - val_categorical_accuracy: 0.3400 - lr: 1.0000e-04\n",
      "Epoch 4/5\n",
      "23/23 [==============================] - ETA: 0s - loss: 133.1504 - categorical_accuracy: 0.5068\n",
      "Epoch 00004: val_loss improved from 140.69827 to 125.04298, saving model to Conv3D_2024-12-1002_48_49.316964/model-00004-133.15044-0.50679-125.04298-0.41000.h5\n",
      "23/23 [==============================] - 145s 7s/step - loss: 133.1504 - categorical_accuracy: 0.5068 - val_loss: 125.0430 - val_categorical_accuracy: 0.4100 - lr: 1.0000e-04\n",
      "Epoch 5/5\n",
      "23/23 [==============================] - ETA: 0s - loss: 118.2519 - categorical_accuracy: 0.5339\n",
      "Epoch 00005: val_loss improved from 125.04298 to 110.99523, saving model to Conv3D_2024-12-1002_48_49.316964/model-00005-118.25192-0.53394-110.99523-0.43000.h5\n",
      "23/23 [==============================] - 151s 7s/step - loss: 118.2519 - categorical_accuracy: 0.5339 - val_loss: 110.9952 - val_categorical_accuracy: 0.4300 - lr: 1.0000e-04\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f7c3bf64040>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model6.fit(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}