PyTorch Tutorial-torch-autograd-differentiation engine.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cab4753a",
   "metadata": {},
   "source": [
    "# PyTorch Tutorial, Differentiation engine:\n",
    "#### Torch.autograd is the built-in differentiation engine \n",
    "- Compute partial derivatives for parameters with **requires_grad=True**\n",
    "- Disable gradient tracking\n",
    "- Use **loss.backward()** for gradient computation\n",
    "- A taste of **gradient descent** and its effect on loss value\n",
    "\n",
    "https://github.com/ostad-ai/PyTorch-Tutorial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "75354f9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# importing the necessary modules\n",
    "import torch\n",
    "import torch.nn as tnn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c5e5ce0",
   "metadata": {},
   "source": [
    "In previous post, we show how to create our neural network. A neural network contains parameters that need to be trained for the specific task. The widely used algorithm for training neural networks is **backpropagation**. To use backpropagation, we need to compute the **gradient** of the loss function with respect to each parameter of the neural network. For this purpose, we need a differentiation engine."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e853c51",
   "metadata": {},
   "source": [
    "For the parameters that need to be optimized, we need to compute the gradients the of loss function with respect to those parameters. In PyTorch, we set the **requires_grad** properties of the parameters to **True** to inform PyTorch to compute gradients for the parameters.\n",
    "<br> In the following, we create a simple linear model for predicting output $y$ from input vector $[x_0,x_1]$\n",
    "<br> $y=b+w_0x_0+w_1x_1$\n",
    "<br>We want to find bias b and weights $w_0$ and $w_1$ so that the loss function below is minimized. For simplicity, we use instantenuous loss for *n*th sample:\n",
    "<br>$loss=(d_n-y_n)^2$\n",
    "<br>We can compute the partial derivatives of loss function with respect to the parameters \n",
    "<br>$\\frac{\\partial loss}{\\partial w_0}$, $\\frac{\\partial loss}{\\partial w_1}$,\n",
    "$\\frac{\\partial loss}{\\partial b}$\n",
    "<br>$\\frac{\\partial loss}{\\partial w_i}=-2(d_n-y_n)x_i$, $i=0,1$\n",
    "<br>$\\frac{\\partial loss}{\\partial b}=-2(d_n-y_n)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5ae0fc7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# the direct method to compute partial derivatives of the loss function above\n",
    "def compute_grads(x,d,w,b):\n",
    "    y=torch.matmul(x,w.detach())+b.detach()\n",
    "    Dw=-2*(d-y)*x\n",
    "    Db=-2*(d-y)\n",
    "    return Dw,Db"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47462aff",
   "metadata": {},
   "source": [
    "Let partial derivatives be computed automatically with torch.autograd as shown below. \n",
    "<br>We compare the results wth the direct method defined above.\n",
    "<Br> In PyTorch, we use **loss.backward** to compute gradients for parameters with **requires_grad=True**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9ba42428",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gradients with autograd: tensor([15.3466, 25.5777]),tensor([5.1155])\n",
      "Gradients with direct method: (tensor([15.3466, 25.5777]), tensor([5.1155]))\n"
     ]
    }
   ],
   "source": [
    "# y=b+w0x0+w1x1 with loss=(d-y)^2\n",
    "x=torch.tensor([3.,5.]) #input vector\n",
    "d=torch.tensor(.2) # desired value\n",
    "w=torch.randn(2,requires_grad=True) # weights\n",
    "b=torch.randn(1,requires_grad=True) #bias\n",
    "y=torch.matmul(x,w)+b # output of model to input x\n",
    "loss=(d-y)**2  # the dieal value is zero, the smaller the better\n",
    "loss.backward()\n",
    "print(f'Gradients with autograd: {w.grad},{b.grad}')\n",
    "print(f'Gradients with direct method: {compute_grads(x,d,w,b)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2626bd63",
   "metadata": {},
   "source": [
    "How to **disable gradient tracking** and why?\n",
    "We said that tensors with requires_grad=True support gradient computation. But, there are times we want to disable gradient tracking.\n",
    "<br>For example, when we want to evaluate or use a trained neural network, we don't need gradient computation. \n",
    "<br>Also, when we want to **freeze** some parameters during training of a neural netowrk, we should use gradient disabling for them. \n",
    "<br>We may disable gradient tracking with:\n",
    " - torch.no_grad()\n",
    " - detach()\n",
    "\n",
    "However, we have the option **requires_grad=False** for tensors to disable gradient computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "dad868ad",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "requires_grad is True\n",
      "requires_grad is False\n",
      "requires_grad is False\n"
     ]
    }
   ],
   "source": [
    "y=torch.matmul(x,w)+b\n",
    "print(f'requires_grad is {y.requires_grad}')\n",
    "with torch.no_grad():\n",
    "    y=torch.matmul(x,w)+b\n",
    "print(f'requires_grad is {y.requires_grad}')\n",
    "y=torch.matmul(x,w)+b\n",
    "y=y.detach()\n",
    "print(f'requires_grad is {y.requires_grad}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c9bede5",
   "metadata": {},
   "source": [
    "**Hint:** Every time we call **loss.backward()**, the gradient values are added to the *grad* propertiy of all leaf nodes of computational graph. Therefore, a second call to loss.backward() leads to gradient accumulation. To get the same result, we should zero out the *grad* property of parameters.\n",
    "<br> In the example below, we check the loss.backward() and its effect on gradients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "354a4b66",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First call to loss.backward():\n",
      "weights and bias grads:\n",
      " tensor([[  0.6597,  -0.6459,   0.6203],\n",
      "        [-12.6338,  12.3691, -11.8791],\n",
      "        [  0.1674,  -0.1639,   0.1574],\n",
      "        [-13.2970,  13.0185, -12.5027],\n",
      "        [  5.7151,  -5.5954,   5.3737]]) tensor([-6.2246,  6.0942, -5.8527])\n",
      "Second call to loss.backward():\n",
      "weights and bias grads:\n",
      " tensor([[  1.3194,  -1.2918,   1.2406],\n",
      "        [-25.2676,  24.7383, -23.7581],\n",
      "        [  0.3348,  -0.3278,   0.3148],\n",
      "        [-26.5941,  26.0370, -25.0054],\n",
      "        [ 11.4302, -11.1907,  10.7474]]) tensor([-12.4492,  12.1884, -11.7055])\n",
      "Third call to loss.backward() with zeroing gradients:\n",
      "weights and bias grads:\n",
      " tensor([[  0.6597,  -0.6459,   0.6203],\n",
      "        [-12.6338,  12.3691, -11.8791],\n",
      "        [  0.1674,  -0.1639,   0.1574],\n",
      "        [-13.2970,  13.0185, -12.5027],\n",
      "        [  5.7151,  -5.5954,   5.3737]]) tensor([-6.2246,  6.0942, -5.8527])\n",
      "--------------\n",
      "The first call and third call have the same values of gradients\n"
     ]
    }
   ],
   "source": [
    "xv=torch.randn(5)\n",
    "dv=torch.ones(3)\n",
    "wv=torch.randn(5,3,requires_grad=True)\n",
    "bv=torch.randn(3,requires_grad=True)\n",
    "yv=torch.matmul(xv,wv)+bv\n",
    "loss=(dv-yv).pow(2).sum()\n",
    "loss.backward(retain_graph=True) #compute gradients of parameters\n",
    "print(f'First call to loss.backward():')\n",
    "print(f'weights and bias grads:\\n {wv.grad} {bv.grad}')\n",
    "loss.backward(retain_graph=True)\n",
    "print(f'Second call to loss.backward():')\n",
    "print(f'weights and bias grads:\\n {wv.grad} {bv.grad}')\n",
    "# making gradients zero before loss.backward()\n",
    "wv.grad.zero_(); bv.grad.zero_()\n",
    "loss.backward()\n",
    "print(f'Third call to loss.backward() with zeroing gradients:')\n",
    "print(f'weights and bias grads:\\n {wv.grad} {bv.grad}')\n",
    "print('--------------')\n",
    "print('The first call and third call have the same values of gradients')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c9c4fc5",
   "metadata": {},
   "source": [
    "A taste of **gradient descent**: Changing parameters with negative of gradients "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e19b3c81",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss before gradient descent: 12.306927680969238\n",
      "Loss after gradient descent: 8.528040885925293\n"
     ]
    }
   ],
   "source": [
    "xv=torch.randn(5) # input\n",
    "dv=torch.ones(3)  # desired output\n",
    "wv=torch.randn(5,3,requires_grad=True) # paramter\n",
    "bv=torch.randn(3,requires_grad=True)   # paramter\n",
    "yv=torch.matmul(xv,wv)+bv\n",
    "loss=(dv-yv).pow(2).sum()\n",
    "print(f'Loss before gradient descent: {loss}')\n",
    "# computing gradients for parameters\n",
    "loss.backward()\n",
    "# gradient descent step\n",
    "with torch.no_grad():\n",
    "    wv-=.05*wv.grad\n",
    "    bv-=.05*bv.grad\n",
    "yv=torch.matmul(xv,wv)+bv\n",
    "loss=(dv-yv).pow(2).sum()\n",
    "print(f'Loss after gradient descent: {loss}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "367566fd",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}