03-feature-modelling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f70149ba",
   "metadata": {},
   "source": [
    "# 3 - Feature Modelling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "429242dc",
   "metadata": {},
   "source": [
    "This is the third notebook of our project. In this notebook we will start applying feature engineering to our datasets in order to preapare them for our object detection models using MXNet.\n",
    "\n",
    "We will implement the framework MXNet as is a proven and efficient deep learning framework optimized for scalability and cross-platform deployment, making it particularly suitable for object detection tasks on AWS. Being natively supported by AWS, MXNet ensures seamless integration, optimal performance, and efficient resource utilization. Furthermore, our project involves meticulous feature engineering to tailor our dataset for the specific requirements of our model. We will transform our dataset into the LST Label format, which is a compact and efficient representation encapsulating image index, variable-length label, and image path. This step ensures that our data is both consistent and optimized for training within the MXNet environment.\n",
    "\n",
    "The format of LST file is:\n",
    "\n",
    "\n",
    "<code>integer_image_index \\t label_of_variable_length \\t relative_path_to_image</code>\n",
    "\n",
    "\n",
    "More info can be found <a href=\"https://cv.gluon.ai/build/examples_datasets/detection_custom.html#lst-label-for-gluoncv-and-mxnet\">here</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cdd241f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import shutil\n",
    "import glob\n",
    "from utils.data_eda_viz_preprocessing import load_csv_as_dataset\n",
    "from utils.data_preprocess import save_datasets_to_csv\n",
    "from utils.data_eda_viz_preprocessing import extract_zip_to_folder\n",
    "from utils.data_engineering import insert_column\n",
    "\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60724cdd",
   "metadata": {},
   "source": [
    "### Loading Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "35f74688",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loading our datasets\n",
    "train = load_csv_as_dataset('data/csv/train.csv')\n",
    "test = load_csv_as_dataset('data/csv/test.csv')\n",
    "\n",
    "# Make copies in order to keep the original datasets\n",
    "train_df = train.copy()\n",
    "test_df = test.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3066331a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Unnamed: 0', 'ImageID', 'Source', 'LabelName', 'Confidence', 'XMin',\n",
       "       'XMax', 'YMin', 'YMax', 'IsOccluded', 'IsTruncated', 'IsGroupOf',\n",
       "       'IsDepiction', 'IsInside', 'XClick1X', 'XClick2X', 'XClick3X',\n",
       "       'XClick4X', 'XClick1Y', 'XClick2Y', 'XClick3Y', 'XClick4Y'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "27fad914",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Unnamed: 0', 'ImageID', 'Source', 'LabelName', 'Confidence', 'XMin',\n",
       "       'XMax', 'YMin', 'YMax', 'IsOccluded', 'IsTruncated', 'IsGroupOf',\n",
       "       'IsDepiction', 'IsInside', 'XClick1X', 'XClick2X', 'XClick3X',\n",
       "       'XClick4X', 'XClick1Y', 'XClick2Y', 'XClick3Y', 'XClick4Y'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e0e5b4f",
   "metadata": {},
   "source": [
    "### Selecting necessary columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e918a465",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = train_df[['LabelName', 'XMin','YMin', 'XMax', 'YMax', 'ImageID']]\n",
    "test_df = test_df[['LabelName', 'XMin','YMin', 'XMax', 'YMax', 'ImageID']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d547311b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LabelName</th>\n",
       "      <th>XMin</th>\n",
       "      <th>YMin</th>\n",
       "      <th>XMax</th>\n",
       "      <th>YMax</th>\n",
       "      <th>ImageID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>/m/0cmf2</td>\n",
       "      <td>0.537500</td>\n",
       "      <td>0.347092</td>\n",
       "      <td>0.998750</td>\n",
       "      <td>0.574109</td>\n",
       "      <td>0644da39dd206abc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>/m/0cmf2</td>\n",
       "      <td>0.286875</td>\n",
       "      <td>0.628385</td>\n",
       "      <td>0.336875</td>\n",
       "      <td>0.652661</td>\n",
       "      <td>01f0114cacd689a3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  LabelName      XMin      YMin      XMax      YMax           ImageID\n",
       "0  /m/0cmf2  0.537500  0.347092  0.998750  0.574109  0644da39dd206abc\n",
       "1  /m/0cmf2  0.286875  0.628385  0.336875  0.652661  01f0114cacd689a3"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9cb2460a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LabelName</th>\n",
       "      <th>XMin</th>\n",
       "      <th>YMin</th>\n",
       "      <th>XMax</th>\n",
       "      <th>YMax</th>\n",
       "      <th>ImageID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>/m/0cmf2</td>\n",
       "      <td>0.444653</td>\n",
       "      <td>0.493125</td>\n",
       "      <td>0.546904</td>\n",
       "      <td>0.585000</td>\n",
       "      <td>0289cd0483d2f758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>/m/0cmf2</td>\n",
       "      <td>0.553125</td>\n",
       "      <td>0.487395</td>\n",
       "      <td>0.632500</td>\n",
       "      <td>0.612512</td>\n",
       "      <td>094d697dbc53a9f2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  LabelName      XMin      YMin      XMax      YMax           ImageID\n",
       "0  /m/0cmf2  0.444653  0.493125  0.546904  0.585000  0289cd0483d2f758\n",
       "1  /m/0cmf2  0.553125  0.487395  0.632500  0.612512  094d697dbc53a9f2"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "322b3692",
   "metadata": {},
   "source": [
    "### Inserting <code>header_cols</code> and <code>label_width</code>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0b753509",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = insert_column(train_df,0,\"header_cols\", 2)\n",
    "train_df = insert_column(train_df,1,\"label_width\", 5)\n",
    "\n",
    "test_df = insert_column(test_df,0,\"header_cols\", 2)\n",
    "test_df = insert_column(test_df,1,\"label_width\", 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "7f7d9790",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>header_cols</th>\n",
       "      <th>label_width</th>\n",
       "      <th>LabelName</th>\n",
       "      <th>XMin</th>\n",
       "      <th>YMin</th>\n",
       "      <th>XMax</th>\n",
       "      <th>YMax</th>\n",
       "      <th>ImageID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>/m/0cmf2</td>\n",
       "      <td>0.537500</td>\n",
       "      <td>0.347092</td>\n",
       "      <td>0.998750</td>\n",
       "      <td>0.574109</td>\n",
       "      <td>0644da39dd206abc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>/m/0cmf2</td>\n",
       "      <td>0.286875</td>\n",
       "      <td>0.628385</td>\n",
       "      <td>0.336875</td>\n",
       "      <td>0.652661</td>\n",
       "      <td>01f0114cacd689a3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   header_cols  label_width LabelName      XMin      YMin      XMax      YMax  \\\n",
       "0            2            5  /m/0cmf2  0.537500  0.347092  0.998750  0.574109   \n",
       "1            2            5  /m/0cmf2  0.286875  0.628385  0.336875  0.652661   \n",
       "\n",
       "            ImageID  \n",
       "0  0644da39dd206abc  \n",
       "1  01f0114cacd689a3  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "988268cc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>header_cols</th>\n",
       "      <th>label_width</th>\n",
       "      <th>LabelName</th>\n",
       "      <th>XMin</th>\n",
       "      <th>YMin</th>\n",
       "      <th>XMax</th>\n",
       "      <th>YMax</th>\n",
       "      <th>ImageID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>/m/0cmf2</td>\n",
       "      <td>0.444653</td>\n",
       "      <td>0.493125</td>\n",
       "      <td>0.546904</td>\n",
       "      <td>0.585000</td>\n",
       "      <td>0289cd0483d2f758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>/m/0cmf2</td>\n",
       "      <td>0.553125</td>\n",
       "      <td>0.487395</td>\n",
       "      <td>0.632500</td>\n",
       "      <td>0.612512</td>\n",
       "      <td>094d697dbc53a9f2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   header_cols  label_width LabelName      XMin      YMin      XMax      YMax  \\\n",
       "0            2            5  /m/0cmf2  0.444653  0.493125  0.546904  0.585000   \n",
       "1            2            5  /m/0cmf2  0.553125  0.487395  0.632500  0.612512   \n",
       "\n",
       "            ImageID  \n",
       "0  0289cd0483d2f758  \n",
       "1  094d697dbc53a9f2  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a15b6b0",
   "metadata": {},
   "source": [
    "### Formatting <code>LabelName</code> column\n",
    "\n",
    "We will rename the column to <code>className</code> and change the value to 0.000 as the framework requires it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "06bcfe07",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df.rename(columns={\"LabelName\": \"className\"}, inplace=True)\n",
    "test_df.rename(columns={\"LabelName\": \"className\"}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ae1b7fe1",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df.className = \"0.000\"\n",
    "test_df.className = \"0.000\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "feaa7893",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>header_cols</th>\n",
       "      <th>label_width</th>\n",
       "      <th>className</th>\n",
       "      <th>XMin</th>\n",
       "      <th>YMin</th>\n",
       "      <th>XMax</th>\n",
       "      <th>YMax</th>\n",
       "      <th>ImageID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.537500</td>\n",
       "      <td>0.347092</td>\n",
       "      <td>0.998750</td>\n",
       "      <td>0.574109</td>\n",
       "      <td>0644da39dd206abc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.286875</td>\n",
       "      <td>0.628385</td>\n",
       "      <td>0.336875</td>\n",
       "      <td>0.652661</td>\n",
       "      <td>01f0114cacd689a3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   header_cols  label_width className      XMin      YMin      XMax      YMax  \\\n",
       "0            2            5     0.000  0.537500  0.347092  0.998750  0.574109   \n",
       "1            2            5     0.000  0.286875  0.628385  0.336875  0.652661   \n",
       "\n",
       "            ImageID  \n",
       "0  0644da39dd206abc  \n",
       "1  01f0114cacd689a3  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "7833db83",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>header_cols</th>\n",
       "      <th>label_width</th>\n",
       "      <th>className</th>\n",
       "      <th>XMin</th>\n",
       "      <th>YMin</th>\n",
       "      <th>XMax</th>\n",
       "      <th>YMax</th>\n",
       "      <th>ImageID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.444653</td>\n",
       "      <td>0.493125</td>\n",
       "      <td>0.546904</td>\n",
       "      <td>0.585000</td>\n",
       "      <td>0289cd0483d2f758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.553125</td>\n",
       "      <td>0.487395</td>\n",
       "      <td>0.632500</td>\n",
       "      <td>0.612512</td>\n",
       "      <td>094d697dbc53a9f2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   header_cols  label_width className      XMin      YMin      XMax      YMax  \\\n",
       "0            2            5     0.000  0.444653  0.493125  0.546904  0.585000   \n",
       "1            2            5     0.000  0.553125  0.487395  0.632500  0.612512   \n",
       "\n",
       "            ImageID  \n",
       "0  0289cd0483d2f758  \n",
       "1  094d697dbc53a9f2  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd09e661",
   "metadata": {},
   "source": [
    "### Formatting <code>ImageID</code> column"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63081ac4",
   "metadata": {},
   "source": [
    "Relative path to the images are required, as we have explained, for MXNET. We will create a new folder and move the images there. The relative path will be the new value of <code>ImageID</code> (that we will rename it to <code>ImagePath</code>) and each value will point to this new folder that will contain the images.\n",
    "\n",
    "\n",
    "First of all, we will format the column and values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "60512e64",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change column name\n",
    "train_df.rename(columns={\"ImageID\": \"ImagePath\"}, inplace=True)\n",
    "test_df.rename(columns={\"ImageID\": \"ImagePath\"}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "7c154c8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Format value to point to the new folder\n",
    "train_df.ImagePath = \"airplanes/images/train/\" + train_df.ImagePath + \".jpg\"\n",
    "test_df.ImagePath = \"airplanes/images/test/\" + test_df.ImagePath + \".jpg\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c59d5c9c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>header_cols</th>\n",
       "      <th>label_width</th>\n",
       "      <th>className</th>\n",
       "      <th>XMin</th>\n",
       "      <th>YMin</th>\n",
       "      <th>XMax</th>\n",
       "      <th>YMax</th>\n",
       "      <th>ImagePath</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.537500</td>\n",
       "      <td>0.347092</td>\n",
       "      <td>0.998750</td>\n",
       "      <td>0.574109</td>\n",
       "      <td>airplanes/images/train/0644da39dd206abc.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.286875</td>\n",
       "      <td>0.628385</td>\n",
       "      <td>0.336875</td>\n",
       "      <td>0.652661</td>\n",
       "      <td>airplanes/images/train/01f0114cacd689a3.jpg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   header_cols  label_width className      XMin      YMin      XMax      YMax  \\\n",
       "0            2            5     0.000  0.537500  0.347092  0.998750  0.574109   \n",
       "1            2            5     0.000  0.286875  0.628385  0.336875  0.652661   \n",
       "\n",
       "                                     ImagePath  \n",
       "0  airplanes/images/train/0644da39dd206abc.jpg  \n",
       "1  airplanes/images/train/01f0114cacd689a3.jpg  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "74acb452",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>header_cols</th>\n",
       "      <th>label_width</th>\n",
       "      <th>className</th>\n",
       "      <th>XMin</th>\n",
       "      <th>YMin</th>\n",
       "      <th>XMax</th>\n",
       "      <th>YMax</th>\n",
       "      <th>ImagePath</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.444653</td>\n",
       "      <td>0.493125</td>\n",
       "      <td>0.546904</td>\n",
       "      <td>0.585000</td>\n",
       "      <td>airplanes/images/test/0289cd0483d2f758.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.553125</td>\n",
       "      <td>0.487395</td>\n",
       "      <td>0.632500</td>\n",
       "      <td>0.612512</td>\n",
       "      <td>airplanes/images/test/094d697dbc53a9f2.jpg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   header_cols  label_width className      XMin      YMin      XMax      YMax  \\\n",
       "0            2            5     0.000  0.444653  0.493125  0.546904  0.585000   \n",
       "1            2            5     0.000  0.553125  0.487395  0.632500  0.612512   \n",
       "\n",
       "                                    ImagePath  \n",
       "0  airplanes/images/test/0289cd0483d2f758.jpg  \n",
       "1  airplanes/images/test/094d697dbc53a9f2.jpg  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83d63da9",
   "metadata": {},
   "source": [
    "Now that we have our dataset with the column order and values as required by MXNet framework, we are going to move the images from the <code>unzipped</code> folder to our new ***train*** and ***test*** folders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "68e8895a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'airplanes/images/test/'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Creating our need folder and subfolders\n",
    "shutil.move(\"unzipped/trainImages/train/data\", \"airplanes/images/train/\")\n",
    "shutil.move(\"unzipped/testImages/data\", \"airplanes/images/test/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "6ae1b80c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of images in train folder: 773\n",
      "Number of images in test folder: 260\n"
     ]
    }
   ],
   "source": [
    "# Images count in each folder\n",
    "folder_train = glob.glob(\"airplanes/images/train/*.jpg\")\n",
    "folder_test = glob.glob(\"airplanes/images/test/*.jpg\")\n",
    "\n",
    "count_train_images = len(folder_train)\n",
    "count_test_images = len(folder_test)\n",
    "\n",
    "print(f\"Number of images in train folder: {count_train_images}\")\n",
    "print(f\"Number of images in test folder: {count_test_images}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33152b03",
   "metadata": {},
   "source": [
    "Once we achieve this, we can save our new dataframes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "8462ac26",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training dataset saved to data/processed_csv/train.csv\n",
      "Test dataset saved to data/processed_csv/test.csv\n"
     ]
    }
   ],
   "source": [
    "save_datasets_to_csv(train_df=train_df, test_df=test_df, folder_path= \"data/processed_csv\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}