TensorFlow: Advanced Techniques/C1 Week 1 - Custom Models, Layers, and Loss Functions with TensorFlow .ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Custom Models, Layers, and Loss Functions with TensorFlow \n",
    "### *Week: 1*\n",
    "\n",
    "#### DeepLearningAI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Description\n",
    "\n",
    "* Compare Functional and Sequential APIs, discover new models you can build with the Functional API, and build a model that produces multiple outputs including a Siamese network.\n",
    "* Build custom loss functions (including the contrastive loss function used in a Siamese network) in order to measure how well a model is doing and help your neural network learn from training data. \n",
    "* Build off of existing standard layers to create custom layers for your models, customize a network layer with a lambda layer, understand the differences between them, learn what makes up a custom layer, and explore activation functions. \n",
    "* Build off of existing models to add custom functionality, learn how to define your own custom class instead of using the Functional or Sequential APIs, build models that can be inherited from the TensorFlow Model class, and build a residual network (ResNet) through defining a custom model class. \n",
    "\n",
    "\n",
    "The DeepLearning.AI TensorFlow: Advanced Techniques Specialization introduces the features of TensorFlow that provide learners with more control over their model architecture and tools that help them create and train advanced ML models.  \n",
    "\n",
    "This Specialization is for early and mid-career software and machine learning engineers with a foundational understanding of TensorFlow who are looking to expand their knowledge and skill set by learning advanced TensorFlow features to build powerful models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sequential Model Example\n",
    "\n",
    "```Python\n",
    "seq_model = Sequential([\n",
    "        Flatten(input_shape=(28, 28)),\n",
    "        Dense(128, activation='relu'),\n",
    "        Dense(10, activation='softmax')\n",
    "        ])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Sequential method return the model.\n",
    "* The first layer is **flatten**, it takes a *image* for *28x28* to flat it.\n",
    "* The second layer is **Dense** layer, this layer has 128 dense neuron and each activation function is **relu** function.\n",
    "* The last layer is  **Dense** layer, it has 10 neuron, which means our output has 10 different categories, and it's activation function is **softmax** function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functional API\n",
    "\n",
    "|1 -> |2 -> |3|\n",
    "|--|--|--|\n",
    "|**Input**|**Layers**|**Model**|\n",
    "|Define input to the model.| Define a set of interconnected layers on the input.|Define the model using the input and output layers.|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining the Input\n",
    "\n",
    "```Python\n",
    "from tensorflow.keras.layers import Input\n",
    "\n",
    "...\n",
    "\n",
    "input = Input(shape(28,28)) # Our example is Mnist dataset.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining the layers\n",
    "\n",
    "```Python\n",
    "from tensorflow.keras.layers import Dense, Flatten\n",
    "\n",
    "...\n",
    "\n",
    "x = Flatten()(input)\n",
    "x = Dense(128, activation='relu')(x)\n",
    "predictions = Dense(10, activation='softmax')(x)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining the Model\n",
    "\n",
    "```Python\n",
    "from tensorflow.leras.models import Model\n",
    "\n",
    "...\n",
    "\n",
    "func_model = Model(inputs=input, outputs=predictions)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [The first lab notebook](C1_W1_Lab_1_functional-practice.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Declaring and Stacking layers\n",
    "\n",
    "```Python\n",
    "\n",
    "def build_model_with_functional():\n",
    "    from tensorflow.keras.models import Model\n",
    "    \n",
    "    input_layer = tf.keras.Input(shape=(28, 28))\n",
    "    flatten_layer = tf.keras.layers.Flatten()(input_layer)\n",
    "    first_dense = tf.keras.layers.Dense(128, activation=tf.nn.relu)(flatten_layer) # This layer should follow the flatten_layer\n",
    "    \n",
    "    # Also we could make other stlye like;\n",
    "    \n",
    "    \"\"\"\n",
    "    first_dense = tf.keras.layers.Dense(128, activation=tf.nn.relu)\n",
    "    first_dense(flatten_layer)\n",
    "    \"\"\"\n",
    "    \n",
    "    output_layer = tf.keras.layers.Dense(10, activation=tf.nn.softmax)(first_dense)\n",
    "    func_model = Model(inputs=input_layer, output=output_layer)\n",
    "    # We may have also multiple inputs and multiple outputs\n",
    "    \n",
    "    return func_model\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Branching Models\n",
    "\n",
    "![inception_models](inception_model.png)\n",
    "\n",
    "*It's not a direct linear model from layer to layer.*\n",
    "\n",
    "#### Pseudocode will be like that\n",
    "\n",
    "```Python\n",
    "layer1 = Dense(32)\n",
    "layer2_1 = Dense(32)(layer1)\n",
    "layer2_2 = Dense(32)(layer1)\n",
    "layer2_3 = Dense(32)(layer1)\n",
    "layer2_4 = Dense(32)(layer1)\n",
    "\n",
    "merge = Concatenate([layer2_1, layer2_2, layer2_3, layer2_4])\n",
    "\n",
    "# Multiple inputs and outputs example\n",
    "func_model = Model(inputs=[input1, input2], outputs=[output1, output2])\n",
    "```\n",
    "\n",
    "![layers_image](layer_1_to_4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [*A Simple Guide to the Versions of the Inception Network*](https://towardsdatascience.com/a-simple-guide-to-the-versions-of-the-inception-network-7fc52b863202)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Multi-Output Model\n",
    "\n",
    "[*Energy Effiency Dataset Link*](https://archive.ics.uci.edu/ml/datasets/Energy+efficiency)\n",
    "\n",
    "The data has *8 features* and *2 labels*.\n",
    "\n",
    "|Features|Labels|\n",
    "|--|--|\n",
    "|Relative Compactness|Heating Load|\n",
    "|Surface Area|Cooling Load|\n",
    "|Wall Area||\n",
    "|Roof Area||\n",
    "|Overall Height||\n",
    "|Orientation||\n",
    "|Glazing Area||\n",
    "|Glazing Area Distribution||\n",
    "\n",
    "![Multiple Output](multiple_output.png)\n",
    "\n",
    "```Python\n",
    "input_layer = Input(shape=(len(train.columns),))\n",
    "first_dense = Dense(Units='128', activation='relu')(input_layer)\n",
    "second_dense = Dense(units='128', activation='relu')(first_dense)\n",
    "\n",
    "y1_output = Dense(unit'1', name='y1_output')(second_dense)\n",
    "\n",
    "third_dense = Dense(units='64', activation='relu')(second_dense)\n",
    "\n",
    "y2_output = Dense(units='1', name='y2_output')(third_dense)\n",
    "\n",
    "# Define the model with input layer and a list of output layers\n",
    "model = Model(inputs=input_layer, outputs=[y1_output, y2_output])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [The second lab notebook](C1_W1_Lab_2_multi-output.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Siamese network: a Multiple-Input model\n",
    "\n",
    "![Siamese_network](siamese_network.png)\n",
    "\n",
    "**Desc**: You have two inputs, in this case two input images, which are processed with the two sub-networks that have the same base neural network architecture. We can measure the Euclidean distance between the output vectors of these networks to predict how similar these two input examples are or how different they are. \n",
    "\n",
    "#### *Siamese network references* \n",
    "1. [ Learning a Similarity Metric Discriminatively, with Application to Face Verification (Chopra, Hadsell, & LeCun, 2005)](http://yann.lecun.com/exdb/publis/pdf/chopra-05.pdf) \n",
    "2. [Similarity Learning with (or without) Convolutional Neural Network (Chatterjee & Luo, n.d.)](http://slazebni.cs.illinois.edu/spring17/lec09_similarity.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Coding a Multi-Input Siamese network\n",
    "\n",
    "**Defining the Base Network**\n",
    "\n",
    "```Python\n",
    "def initialize_base_network():\n",
    "    input = Input(shape=(28, 28),)\n",
    "    x = Flatten()(input)\n",
    "    x = Dense(128, activation='relu')(x)\n",
    "    x = Dropout(0.1)\n",
    "    x = Dense(128, activation='relu')(x)\n",
    "    x = Dropout(0.1)\n",
    "    x = Dense(128, activation='relu')(x)\n",
    "    return Model(inputs= input, outputs= x)\n",
    "```\n",
    "\n",
    "### Re-using the base network\n",
    "\n",
    "```Python\n",
    "input_a = Input(shape(28, 28),)\n",
    "input_b = Input(shape(28, 28),)\n",
    "\n",
    "vect_output_a = base_network(input_a)\n",
    "vect_output_b = base_network(input_b)\n",
    "```\n",
    "\n",
    "![Multi_input](siamese_multi_input.png)\n",
    "\n",
    "### Output functions\n",
    "\n",
    "```Python\n",
    "def euclidean_disance(vects):\n",
    "    x, y = vects\n",
    "    sum_square = K.sum(K.square(x - y), axis= 1, keepdims=True)\n",
    "    return K.sqrt(K.maximum(sum_square, K.epsilon()))\n",
    "\n",
    "\n",
    "def eucl_dist_output_shape(shapes):\n",
    "    shape1, shape2 = shapes\n",
    "    return(shape1[0], 1)\n",
    "\n",
    "\n",
    "output = Lambda(euclidean_distance, output_shape = eucl_dist_output_shape)([vec_output_a, vect?output_b])\n",
    "\n",
    "```\n",
    "\n",
    "### Defining the final model\n",
    "\n",
    "```Python\n",
    "model = Model([input_a, input_b], output)\n",
    "\n",
    "rms = RMSprop()\n",
    "model.compile(loss=contrastive_loss, optimizer=rms) # contrastive_loss is custom loss function. It's created for our model.\n",
    "```\n",
    "\n",
    "\n",
    "### Train the Model\n",
    "\n",
    "```Python\n",
    "model.fit([tr_pairs[:,0], tr_pairs[:,1]], tr_y # Training data -> tr\n",
    "           epochs=20,\n",
    "          batch_size=128,\n",
    "          validation_data=([ts_pairs[:,0], ts_pairs[:,1]], rs_y)\n",
    "         )\n",
    "```\n",
    "\n",
    "**tr_pairs[:,0]** feeds the first, left column input of our model.\n",
    "\n",
    "**tr_pairs[:,1]** feeds the seconds, right column input of our model.\n",
    "\n",
    "**tr_y** is the similarity value, 1 is similar and 0 is different each other.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![similarity_output](similarity.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [The third lab notebook](C1_W1_Lab_3_siamese-network.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### [Serkan UYSAL](https://github.com/uysalserkan)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}