TensorFlow: Advanced Techniques/C1_W2_Lab_1_huber-loss.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ungraded Lab: Huber Loss\n",
    "\n",
    "In this lab, we'll walk through how to create custom loss functions. In particular, we'll code the [Huber Loss](https://en.wikipedia.org/wiki/Huber_loss) and use that in training the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0p84I7yFHRT2"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "  # %tensorflow_version only exists in Colab.\n",
    "  %tensorflow_version 2.x\n",
    "except Exception:\n",
    "  pass\n",
    "\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from tensorflow import keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare the Data\n",
    "\n",
    "Our dummy dataset is just a pair of arrays `xs` and `ys` defined by the relationship $y = 2x - 1$. `xs` are the inputs while `ys` are the labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inputs\n",
    "xs = np.array([-1.0,  0.0, 1.0, 2.0, 3.0, 4.0], dtype=float)\n",
    "\n",
    "# labels\n",
    "ys = np.array([-3.0, -1.0, 1.0, 3.0, 5.0, 7.0], dtype=float)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the model\n",
    "\n",
    "Let's build a simple model and train using a built-in loss function like the `mean_squared_error`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2eY7fw0EHwda"
   },
   "outputs": [],
   "source": [
    "model = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])])\n",
    "model.compile(optimizer='sgd', loss='mean_squared_error')\n",
    "model.fit(xs, ys, epochs=500,verbose=0)\n",
    "\n",
    "print(model.predict([10.0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom Loss\n",
    "\n",
    "Now let's see how we can use a custom loss. We first define a function that accepts the ground truth labels (`y_true`) and model predictions (`y_pred`) as parameters. We then compute and return the loss value in the function definition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "fHtDxE0lI_Hg"
   },
   "outputs": [],
   "source": [
    "def my_huber_loss(y_true, y_pred):\n",
    "    threshold = 1\n",
    "    error = y_true - y_pred\n",
    "    is_small_error = tf.abs(error) <= threshold\n",
    "    small_error_loss = tf.square(error) / 2\n",
    "    big_error_loss = threshold * (tf.abs(error) - (0.5 * threshold))\n",
    "    return tf.where(is_small_error, small_error_loss, big_error_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the loss function is as simple as specifying the loss function in the `loss` argument of `model.compile()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "K7rgmb5qH5QX"
   },
   "outputs": [],
   "source": [
    "model = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])])\n",
    "model.compile(optimizer='sgd', loss=my_huber_loss)\n",
    "model.fit(xs, ys, epochs=500,verbose=0)\n",
    "print(model.predict([10.0]))"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "include_colab_link": true,
   "name": "first-loss.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}