PyTorch Tutorial-torch-nn module for neural network definition.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cab4753a",
   "metadata": {},
   "source": [
    "# PyTorch Tutorial, Model (neural network) definition:\n",
    "#### Torch.nn is used for model (neural network) definition \n",
    "- Download the MNIST dataset\n",
    "- Convert the labels into one-hot vectors\n",
    "- Use DataLoader on the downloaded MNIST dataset\n",
    "- Define componenets of a neural network to deal with the MNIST\n",
    "- Define the neural network with **Sequential** or in a *class* with **forward** method\n",
    "\n",
    "https://github.com/ostad-ai/PyTorch-Tutorial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "75354f9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# importing the necessary modules\n",
    "import torch\n",
    "import torch.nn as tnn\n",
    "from torchvision import datasets\n",
    "from torchvision.transforms import ToTensor, Lambda\n",
    "from torch.utils.data import DataLoader"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c5e5ce0",
   "metadata": {},
   "source": [
    "We download the MNIST dataset in both training_set and test_set, each contains pairs of (features,label). We have seen this in post of datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9ba42428",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_set=datasets.MNIST(root='./MNIST',train=True, download=True, transform=ToTensor())\n",
    "test_set=datasets.MNIST(root='./MNIST',train=False,download=True,transform=ToTensor())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75ea41d8",
   "metadata": {},
   "source": [
    "The **MNIST dataset** as shown below holds 60000 samples for training, and 10000 samples for testing.\n",
    "Each sample is composed of an image with size 28-by-28, and label with integer value from set {0,1,2...,9}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1ab96309",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training dataset:\n",
      "Size of features: torch.Size([60000, 28, 28])\n",
      "Size of labels: torch.Size([60000])\n",
      "The class labels are from set: tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])\n",
      "The number of classes:10\n",
      "Test dataset:\n",
      "Size of features: torch.Size([10000, 28, 28])\n",
      "Size of labels: torch.Size([10000])\n"
     ]
    }
   ],
   "source": [
    "print('Training dataset:')\n",
    "print(f'Size of features: {training_set.data.size()}')\n",
    "print(f'Size of labels: {training_set.targets.size()}')\n",
    "print(f'The class labels are from set: {training_set.targets.unique()}')\n",
    "Nclasses=training_set.targets.unique().size().numel()\n",
    "print(f'The number of classes:{Nclasses}')\n",
    "print('Test dataset:')\n",
    "print(f'Size of features: {test_set.data.size()}')\n",
    "print(f'Size of labels: {test_set.targets.size()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d88bc2b",
   "metadata": {},
   "source": [
    "We saw in previous post that for classification problems, the labels of samples should be in form one-hot vectors.\n",
    "<br>Therefore, we convert labels into one-hot vectors by assigning the **target_transform** of  datasets using one-hot-transform as defined below "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "578d1e90",
   "metadata": {},
   "outputs": [],
   "source": [
    "one_hot_transform=Lambda(lambda y:torch.zeros(Nclasses,dtype=torch.float).index_put([torch.tensor(y)],\n",
    "                                                                         values=torch.tensor(1.)))\n",
    "training_set.target_transform=one_hot_transform\n",
    "test_set.target_transform=one_hot_transform"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "245726d9",
   "metadata": {},
   "source": [
    "Checking that one-hot vectors are created by above target-transform:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ad62bb85",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label of sample 23664 is: 4\n",
      "One-hot vector of label above: tensor([0., 0., 0., 0., 1., 0., 0., 0., 0., 0.])\n"
     ]
    }
   ],
   "source": [
    "index=torch.randint(len(training_set),(1,))[0].item()\n",
    "_,y=training_set[index]\n",
    "print(f'Label of sample {index} is: {training_set.targets[index]}')\n",
    "print(f'One-hot vector of label above: {y}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4401921",
   "metadata": {},
   "source": [
    "Now, we use **DataLoader** to make the dataset iterable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "24f5f95a",
   "metadata": {},
   "outputs": [],
   "source": [
    "batchSize=8\n",
    "train_dataloader=DataLoader(training_set,batch_size=batchSize,shuffle=True)\n",
    "test_dataloader=DataLoader(test_set,batch_size=batchSize,shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a30ad39",
   "metadata": {},
   "source": [
    "Checking how a batch of dataset with dataloader is retreived:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d5b6608a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of a batch=8 of features: torch.Size([8, 1, 28, 28])\n",
      "The number of channels of input features: 1\n",
      "Size of a batch=8 of labels: torch.Size([8, 10])\n"
     ]
    }
   ],
   "source": [
    "# just one step in for-loop over train_dataloader\n",
    "for x,y in train_dataloader:\n",
    "    print(f'Size of a batch={batchSize} of features: {x.size()}')\n",
    "    print(f'The number of channels of input features: {x.size()[1]}')\n",
    "    print(f'Size of a batch={batchSize} of labels: {y.size()}')\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9897e686",
   "metadata": {},
   "source": [
    "For processing images of for (C,H,W), we may use convolutional layers. For example Conv2d(cin,cout,kernel_size,...)\n",
    "<br> In the example below, a Conv2d layer is created of kernel-size=3, and cin=1, cout=32. Let's see how a batch of input pairs are changed in size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f8ec45d3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We have a batch=8 of feature vectors having size: [8, 1, 28, 28]\n",
      "The conv2d output of a batch=8 of feature vectors has the size: [8, 1, 28, 28]\n"
     ]
    }
   ],
   "source": [
    "cin,cout,kernelSize=1,32,3\n",
    "conv1=tnn.Conv2d(cin,cout,kernel_size=kernelSize)\n",
    "out=conv1(x)\n",
    "print(f'We have a batch={batchSize} of feature vectors having size: {list(x.size())}')\n",
    "print(f'The conv2d output of a batch={batchSize} of feature vectors has the size: {list(x.size())}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1dd7df48",
   "metadata": {},
   "source": [
    "The convolutional layer has parameters we call them **weight** and **bias**. \n",
    "<br>Let's see the size of these paramters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "880b181c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Weight size of the conv2d layer: [32, 1, 3, 3]\n",
      "Bias size of the conv2d layer: 32\n"
     ]
    }
   ],
   "source": [
    "print(f'Weight size of the conv2d layer: {list(conv1.weight.size())}')\n",
    "print(f'Bias size of the conv2d layer: {conv1.bias.size().numel()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32e1f452",
   "metadata": {},
   "source": [
    "We usually use a maxpooling layer after a convolutional layer.\n",
    "<br>We may use **MaxPool2d**, which is applied in the example below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "494d71de",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([8, 32, 13, 13])\n"
     ]
    }
   ],
   "source": [
    "maxpool1=tnn.MaxPool2d(kernel_size=2,stride=2)\n",
    "out2=maxpool1(out)\n",
    "print(out2.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a3861f9",
   "metadata": {},
   "source": [
    "We may sometimes need to flatten the output of layer, especially a convolutional layer. \n",
    "<br>For this purpose, we may use **Flatten()**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f68b59ba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([8, 5408])\n"
     ]
    }
   ],
   "source": [
    "flatten1=tnn.Flatten()\n",
    "out3=flatten1(out2)\n",
    "print(out3.size()) # which is 32*13*13"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2f5d100",
   "metadata": {},
   "source": [
    "A linear layer is composed of a number of neurons. For using such a layer, **Linear** is used. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1fda0f3e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([8, 10])\n"
     ]
    }
   ],
   "source": [
    "linear1=tnn.Linear(5408,Nclasses)\n",
    "out4=linear1(out3)\n",
    "print(out4.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33c71375",
   "metadata": {},
   "source": [
    "In neural networks, we often use activation functions, which produce nonlienarity in the network.\n",
    "<br> **ReLU** and **Softmax** are two examples of activation functions. These functions do not change the size of their inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e4e4fc24",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([8, 10])\n"
     ]
    }
   ],
   "source": [
    "act1=tnn.Softmax(dim=1)\n",
    "out5=act1(out4)\n",
    "print(out5.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "500353ba",
   "metadata": {},
   "source": [
    "With the components introduced so far, we are able to define a neural network. \n",
    "<br>On way is to use **Sequential**. \n",
    "<br>The components of a *neural network* may be defined in *Sequential*, where the data is passed through them with the same order of components as defined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f8458610",
   "metadata": {},
   "outputs": [],
   "source": [
    "model=tnn.Sequential(\n",
    "      tnn.Conv2d(cin,cout,kernel_size=kernelSize),\n",
    "      tnn.ReLU(),\n",
    "      tnn.MaxPool2d(kernel_size=2,stride=2),\n",
    "      tnn.Flatten(),\n",
    "      tnn.Linear(5408,Nclasses),\n",
    "      tnn.Softmax(dim=1)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5ded655",
   "metadata": {},
   "source": [
    "The model gets the features part of each sample, and produces the output.\n",
    "<br>Now, we see the size of a batch of input vectors, versus the size of a batch of output vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "0f56321a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The size of input batch of features: [8, 1, 28, 28]\n",
      "The size of output batch: [8, 10]\n"
     ]
    }
   ],
   "source": [
    "y_out=model(x)\n",
    "print(f'The size of input batch of features: {list(x.size())}')\n",
    "print(f'The size of output batch: {list(y_out.size())}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41d24320",
   "metadata": {},
   "source": [
    "We can also define our neural netowk in a class inherited from torch.nn.Module. This class must have __init__() and forward() methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "959b4b89",
   "metadata": {},
   "outputs": [],
   "source": [
    "class NNet(tnn.Module):\n",
    "    def __init__(self,cin=1,cout=32,kernelSize=3,Nclasses=10):\n",
    "        super().__init__()\n",
    "        self.conv1=tnn.Conv2d(cin,cout,kernel_size=kernelSize)\n",
    "        self.relu=tnn.ReLU()\n",
    "        self.pool1=tnn.MaxPool2d(kernel_size=2,stride=2)\n",
    "        self.flatten=tnn.Flatten()\n",
    "        self.fc1=tnn.Linear(5408,Nclasses)\n",
    "        self.softmax=tnn.Softmax(dim=1)\n",
    "    def forward(self,x):\n",
    "        return self.softmax(self.fc1(self.flatten(self.pool1(self.relu(self.conv1(x))))))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "053c20ad",
   "metadata": {},
   "source": [
    "Applying the defined **NNet** to a batch of input vectors is the same as using **Sequential** mentioned earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "6f2de7ae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The size of input batch of features: [8, 1, 28, 28]\n",
      "The size of output batch: [8, 10]\n"
     ]
    }
   ],
   "source": [
    "model2=NNet()\n",
    "y_out2=model2(x)\n",
    "print(f'The size of input batch of features: {list(x.size())}')\n",
    "print(f'The size of output batch: {list(y_out2.size())}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dad868ad",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}