Notebook_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "caf1eb28",
   "metadata": {},
   "source": [
    "![logo](images/Makeathon_Logo.png)\n",
    "# Notebook 1 - Introduction & Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2214e65",
   "metadata": {},
   "source": [
    "In this Notebook we will give you a short introduction into transfer learning and an overview of the remaining three notebooks which will cover different topics in AI.\n",
    "\n",
    "##### Structure\n",
    "1. Transfer Learning\n",
    "2. Natural Language Processing - Notebook 2\n",
    "3. Computer Vision - Notebook 3\n",
    "4. Regression - Notebook 4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "957676d9",
   "metadata": {},
   "source": [
    "## 1. Transfer Learning\n",
    "The idea of transfer learning is to leverage the knowledge of another task for the task of interest. \n",
    "\n",
    "##### Definition:\n",
    "\n",
    "Given a source domain ${D_S}$ and a learning task ${T_S}$, a target domain ${D_T}$ and learning task ${T_T}$, transfer learning aims to help improve the learning of the target predictive function ${f_T (*)}$ using the knowledge in ${D_S}$ and ${T_S}$, where ${D_S}$ $\\neq$ ${D_T}$, or ${T_S}$ $\\neq$ ${T_T}$. (http://www-edlab.cs.umass.edu/cs689/reading/transfer-learning.pdf)\n",
    "\n",
    "\n",
    "\n",
    "In deep learning we can levarage feature representations from a pre-trained model, so we don’t have to train a new model from scratch. Since pre-trained models are usually trained on huge datasets we assume that the model will learn generic feature representations which will enable the model to quickly generalize from only few examples in the new dataset. This can be best illustared for visual categories which share low-levelnotions of edges and visual shapes, the eﬀects of geometric changes, changes inlighting, and so on. \n",
    "\n",
    "\n",
    "<div>\n",
    "<img src=\"images/transfer_learning.png\" width=\"800\">\n",
    "</div>\n",
    "\n",
    "\n",
    "https://theaisummer.com/medical-imaging-transfer-learning/\n",
    "\n",
    "\n",
    "##### The two major transfer learning scenarios look as follows:\n",
    "\n",
    "- Fine-tune the network: \n",
    "    The entire pre-trained model is trained again on the dataset of interest. In this case, the error is back-propagated through the entire architecture and the pre-trained weights of the model are updated based on the new dataset.\n",
    "- Freeze some layers: Only parts of the network are trained, the frozen layers won't be updated during the training. A common scenario in computer vision architectures is to freeze the convolutional feature extractor and only train the classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "132bcdec",
   "metadata": {},
   "source": [
    "##### Transfer Learning with PyTorch\n",
    "You can get your pre-trained model from https://pytorch.org/hub/, GitHub repos or other libaries. Often you will have the option to directly download the pre-trained model, but sometimes you will only be able to access the pre-trained parameters which you then need to load into your model (https://pytorch.org/tutorials/beginner/saving_loading_models.html). \n",
    "\n",
    "Once you have the pre-trained model you can access your models parameters by calling <code> model.parameters() or model.named_parameters() </code> (the latter will also return the name of the parameter). For each parameter you can set if it should compute the gradient or not by <code> param.requires_grad=True or param.requires_grad=False </code>, i.e. this controls if the parameter will be updated or not. \n",
    "\n",
    "In most cases you need to replace the head of your network since it doesn't fit your task (e.g. the model was pre-trained with 1000 classes but you only have 4). Note, this is not an automatic procedure and is unique to each model.\n",
    "\n",
    "Let's do an example where we take a pre-trained ResNet18 (1000 classes), freeze the feature extractor and replace the classification head so it has a output of 4 classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "551c7c7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchvision\n",
    "\n",
    "model = torchvision.models.resnet18(pretrained=True) #Download pre-trained model\n",
    "\n",
    "for param in model.parameters(): #Freeze ALL Layers (doesn't matter that we freeze the classifier, since we will replace it in the next step)\n",
    "    param.requires_grad = False\n",
    "\n",
    "num_ftrs = model.fc.in_features #Get the number of input features for the classifier -> This is the part which is unique to every model\n",
    "model.fc = torch.nn.Linear(num_ftrs, 4) #Replace classifier with new one"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc98c36a",
   "metadata": {},
   "source": [
    "\n",
    "##### Useful Resources:\n",
    "- https://cs231n.github.io/transfer-learning/\n",
    "- https://www.deeplearningbook.org/contents/representation.html\n",
    "- https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html\n",
    "- https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html\n",
    "- https://towardsdatascience.com/a-comprehensive-hands-on-guide-to-transfer-learning-with-real-world-applications-in-deep-learning-212bf3b2f27a\n",
    "- https://medium.com/starschema-blog/transfer-learning-the-dos-and-donts-165729d66625"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b53622be",
   "metadata": {},
   "source": [
    "## 2. Natural language processing - Notebook 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19535cc3",
   "metadata": {},
   "source": [
    "Notebook 2 is all about Natural Language Processing. We've got three use cases for you that hopefully can serve as a starting point / inspiration for your own app!\n",
    "\n",
    "We'll cover:\n",
    "* Information extract\n",
    "* Text classification\n",
    "* Text generation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17b183ff",
   "metadata": {},
   "source": [
    "What you'll get out:\n",
    "* What ML-tools you could use (GPT-2, pretrained embeddings, tokenizers)\n",
    "* What text processing pipelines look like\n",
    "* What UI-tools you could use (Widgets). Granted, those UIs dont look beatiful, but the do the job!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7513ceb0",
   "metadata": {},
   "source": [
    "## 3. Computer Vision - Notebook 3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80b718e3",
   "metadata": {},
   "source": [
    "In Notebook 3 we will cover three use cases : Image classification, semantic segmentation and object detection. But for the sake of completeness there will be also a short section for instance segmentation here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a93d1883",
   "metadata": {},
   "source": [
    "<div>\n",
    "<img src=\"images/cv.jpg\" width=\"1000\">\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5ff7dd2",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Image Classification :\n",
    "In image classification you predict the type or class of an object in an image.\n",
    "\n",
    "###### Output:\n",
    "Image class label.\n",
    "\n",
    "###### Models:\n",
    "- VGG (https://arxiv.org/abs/1409.1556)\n",
    "- ResNet (https://arxiv.org/abs/1512.03385)\n",
    "- Inceptionv3 (https://arxiv.org/abs/1512.00567)\n",
    "- DenseNet (https://arxiv.org/abs/1608.06993)\n",
    "- EfficientNet (https://arxiv.org/pdf/1905.11946.pdf)\n",
    "\n",
    "###### Metrics:\n",
    "- Accuracy\n",
    "- Precision\n",
    "- Recall\n",
    "- F1 Score\n",
    "- AUCRoc\n",
    "\n",
    "###### Useful Resources:\n",
    "- https://neptune.ai/blog/evaluation-metrics-binary-classification\n",
    "- https://viso.ai/computer-vision/image-classification/\n",
    "\n",
    "### Semantic Segmentation :\n",
    "Semantic segmentation is the process of classifying each pixel belonging to a particular label. It doesn't differentiate across different instances of the same object. For example if there are 2 cats in an image, semantic segmentation gives same label to all the pixels of both cats.\n",
    "\n",
    "###### Output:\n",
    "Segmentation map where each pixel contains a class label. \n",
    "\n",
    "###### Models:\n",
    "- FCN (https://arxiv.org/abs/1411.4038)\n",
    "- DeepLab v3 (https://arxiv.org/abs/1706.05587v3)\n",
    "- UNet (https://arxiv.org/abs/1505.04597)\n",
    "\n",
    "###### Metrics:\n",
    "- Pixel Accuracy\n",
    "- IoU (Jaccard Index)\n",
    "- Dice Coefficient (F1 Score)\n",
    "\n",
    "\n",
    "###### Useful Resources\n",
    "- https://www.jeremyjordan.me/semantic-segmentation/\n",
    "- https://towardsdatascience.com/metrics-to-evaluate-your-semantic-segmentation-model-6bcb99639aa2\n",
    "- https://neptune.ai/blog/image-segmentation-in-2020\n",
    "\n",
    "### Object Detection :\n",
    "Object detection locates the presence of objects with a bounding box and types or classes of the located objects in an image. It combines the tasks of object localization + classification. \n",
    "\n",
    "###### Output: \n",
    "Multiple instances of:\n",
    "- Object location (bounding box)\n",
    "- Object class label\n",
    "\n",
    "###### Models:\n",
    "- Faster R-CNN (https://arxiv.org/abs/1506.01497)\n",
    "- YOLOv3 (https://arxiv.org/abs/1804.02767)\n",
    "- SSD (https://arxiv.org/abs/1512.02325)\n",
    "\n",
    "###### Metrics:\n",
    "- Average Precision (AP@.50, AP@.75, ...)\n",
    "- Mean Average Precision (mAP)\n",
    "\n",
    "##### Useful Resources\n",
    "- http://cs231n.stanford.edu/slides/2021/discussion_6_detection.pdf\n",
    "- https://viso.ai/deep-learning/object-detection/\n",
    "- https://jonathan-hui.medium.com/map-mean-average-precision-for-object-detection-45c121a31173\n",
    "- https://github.com/rafaelpadilla/Object-Detection-Metrics#metrics\n",
    "- https://detectron2.readthedocs.io/en/latest/tutorials/getting_started.html\n",
    "\n",
    "### Instance Segmentation :\n",
    "In Instance segmentation you label each foreground pixel with object and instance. It differs from semantic segmentation in the sense that it distinguishes individual objects, in contrast to considering them as a single semantic class. Basically Instance segmentation is object detection + semantic segmentation. As can be seen in the image above all 3 dogs are assigned different colours i.e different labels. With semantic segmentation all of them would have been assigned the same colour/label.\n",
    "\n",
    "###### Output: \n",
    "Multiple instances of:\n",
    "- Object location (bounding box)\n",
    "- Object class label\n",
    "- Object mask\n",
    "\n",
    "###### Models:\n",
    "- Mask R-CNN (https://arxiv.org/abs/1703.06870)\n",
    "- YOLACT (https://arxiv.org/abs/1904.02689)\n",
    "- PANet (https://arxiv.org/abs/1803.01534)\n",
    "\n",
    "###### Metrics:\n",
    "- Average Precision (AP@.50, AP@.75, ...)\n",
    "- Mean Average Precision (mAP)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47ae2ec3",
   "metadata": {},
   "source": [
    "### More Resources:\n",
    "\n",
    "Since transformers are everywhere nowadays it´s worth checking out Vision Transformers (ViT) - https://arxiv.org/abs/2010.11929 / https://viso.ai/deep-learning/vision-transformer-vit/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "612deeb4",
   "metadata": {},
   "source": [
    "## 4. Regression - Notebook 4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "292c4248",
   "metadata": {},
   "source": [
    "Notebook 4 covers regression (prediction of some continuous value, not a class). This notebook is a little shorter compared to Notebooks 2 and 3. Which is not a coincidence: The ML-architectures used are way simpler than those sophisticated transformers and ConvNets covered before. However, this does not impair their power when it comes to regression problems - because this is where those architectures really shine! \n",
    "\n",
    "We'll hat a look at:\n",
    "* Decision trees\n",
    "* Random forests"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ef342e7",
   "metadata": {},
   "source": [
    "What you'll get out:\n",
    "* How to use **sklearn** to grow your decision tree or random forest and make some robust predictions"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}