Deep Learning Fundamentals 11.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0c3366eb",
   "metadata": {
    "id": "0c3366eb"
   },
   "source": [
    "# Deep Learning Fundamentals 11 -  Important Features of Tensorflow"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa2bc935",
   "metadata": {
    "id": "fa2bc935"
   },
   "source": [
    "Welcome to another notebook on Deep Learning Fundamentals. In this notebook, we will be talking about the Data API of Tensorflow which lets us easily manage data in a pipeline, and Tensorflow datasets which provide us with an easy way to download commonly used datasets. Moreover, we will learn more about Preprocessing layers, Embeddings, and TensorflowHub while exploring Tensorflow Data/Datasets. Let's get started."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0ad335a1",
   "metadata": {
    "id": "0ad335a1"
   },
   "outputs": [],
   "source": [
    "import sklearn\n",
    "import numpy as np\n",
    "import os\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c2a4dea",
   "metadata": {
    "id": "6c2a4dea"
   },
   "source": [
    "## The Tensorflow Data API"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0089ab5",
   "metadata": {
    "id": "d0089ab5"
   },
   "source": [
    "In this part, we will have a short case in which we assume we are dealing with a big dataset as if it normally does not fit into memory. However, before getting onto that I would like to show some functions that are highly utilized while using the Data API. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "00c8010e",
   "metadata": {
    "id": "00c8010e"
   },
   "outputs": [],
   "source": [
    "dataset = tf.data.Dataset.range(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "09757aad",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "09757aad",
    "outputId": "d18b73b7-27ee-48ae-e1b7-f073f76b9452"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([0 1 2 3 4 5 6], shape=(7,), dtype=int64)\n",
      "tf.Tensor([7 8 9 0 1 2 3], shape=(7,), dtype=int64)\n",
      "tf.Tensor([4 5 6 7 8 9 0], shape=(7,), dtype=int64)\n",
      "tf.Tensor([1 2 3 4 5 6 7], shape=(7,), dtype=int64)\n",
      "tf.Tensor([8 9], shape=(2,), dtype=int64)\n"
     ]
    }
   ],
   "source": [
    "dataset = dataset.repeat(3).batch(7)\n",
    "for item in dataset:\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efca7c77",
   "metadata": {
    "id": "efca7c77"
   },
   "source": [
    "When we want to transform the instances in our dataset, we can use `map()` and `apply()` methods. They both almost do the same thing, though there is a little difference between them.\n",
    "\n",
    "* While the `map()` method applies a transformation to each item, the `apply()` method applies a transformation to the dataset as a whole. - [Géron, A. (2019)](https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b8a02ef8",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "b8a02ef8",
    "outputId": "cad1a640-1991-4729-fcc9-dd2a27fbfbae"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<MapDataset element_spec=TensorSpec(shape=(None,), dtype=tf.int64, name=None)>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset = dataset.map(lambda x: x * 2)\n",
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c02791b",
   "metadata": {
    "id": "8c02791b"
   },
   "source": [
    "We can also use `filter()` methods for filtering the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "de68df56",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "de68df56",
    "outputId": "123bda10-b44e-475b-a28f-bdda15c45e9e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(0, shape=(), dtype=int64)\n",
      "tf.Tensor(2, shape=(), dtype=int64)\n",
      "tf.Tensor(4, shape=(), dtype=int64)\n"
     ]
    }
   ],
   "source": [
    "dataset = dataset.unbatch()\n",
    "dataset = dataset.filter(lambda x: x < 10)  # keep only items < 10\n",
    "for item in dataset.take(3):\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4235aa9",
   "metadata": {
    "id": "f4235aa9"
   },
   "source": [
    "We can use `shuffle()` method for shuffling our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ea619d32",
   "metadata": {
    "id": "ea619d32"
   },
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "dataset = tf.data.Dataset.range(10).repeat(3)\n",
    "dataset = dataset.shuffle(buffer_size=3, seed=42).batch(7)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a16391e",
   "metadata": {
    "id": "8a16391e"
   },
   "source": [
    "## Creating a pipeline with Tensorflow"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b76131c",
   "metadata": {
    "id": "9b76131c"
   },
   "source": [
    "In this part, I will load the California House Dataset and split it into pieces. I do this as an example for the case when we have a very large dataset that does not fit in memory. In these cases, we can split the data into many files first, then let Tensorflow read these files in parallel."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "063f4939",
   "metadata": {
    "id": "063f4939"
   },
   "source": [
    "Let's load the dataset first and split it into training,test,validation folds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7a8be4e5",
   "metadata": {
    "id": "7a8be4e5"
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_california_housing\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "housing = fetch_california_housing()\n",
    "X_train_full, X_test, y_train_full, y_test = train_test_split(\n",
    "    housing.data, housing.target.reshape(-1, 1), random_state=42)\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(\n",
    "    X_train_full, y_train_full, random_state=42)\n",
    "\n",
    "scaler = StandardScaler()\n",
    "scaler.fit(X_train)\n",
    "X_mean = scaler.mean_\n",
    "X_std = scaler.scale_\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c6d23e0",
   "metadata": {
    "id": "7c6d23e0"
   },
   "source": [
    "Now let's split the dataset into 20 CSV files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d07450a4",
   "metadata": {
    "id": "d07450a4"
   },
   "outputs": [],
   "source": [
    "def save_to_multiple_csv_files(data, name_prefix, header=None, n_parts=10):\n",
    "    housing_dir = os.path.join(\"datasets\", \"housing\")\n",
    "    os.makedirs(housing_dir, exist_ok=True)\n",
    "    path_format = os.path.join(housing_dir, \"my_{}_{:02d}.csv\")\n",
    "\n",
    "    filepaths = []\n",
    "    m = len(data)\n",
    "    for file_idx, row_indices in enumerate(np.array_split(np.arange(m), n_parts)):\n",
    "        part_csv = path_format.format(name_prefix, file_idx)\n",
    "        filepaths.append(part_csv)\n",
    "        with open(part_csv, \"wt\", encoding=\"utf-8\") as f:\n",
    "            if header is not None:\n",
    "                f.write(header)\n",
    "                f.write(\"\\n\")\n",
    "            for row_idx in row_indices:\n",
    "                f.write(\",\".join([repr(col) for col in data[row_idx]]))\n",
    "                f.write(\"\\n\")\n",
    "    return filepaths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1b8b2bec",
   "metadata": {
    "id": "1b8b2bec"
   },
   "outputs": [],
   "source": [
    "train_data = np.c_[X_train, y_train]\n",
    "valid_data = np.c_[X_valid, y_valid]\n",
    "test_data = np.c_[X_test, y_test]\n",
    "header_cols = housing.feature_names + [\"MedianHouseValue\"]\n",
    "header = \",\".join(header_cols)\n",
    "\n",
    "train_filepaths = save_to_multiple_csv_files(train_data, \"train\", header, n_parts=20)\n",
    "valid_filepaths = save_to_multiple_csv_files(valid_data, \"valid\", header, n_parts=10)\n",
    "test_filepaths = save_to_multiple_csv_files(test_data, \"test\", header, n_parts=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86e3a5bc",
   "metadata": {
    "id": "86e3a5bc"
   },
   "source": [
    "Let's look at one of the files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e86952fd",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "e86952fd",
    "outputId": "e8f7be4b-75c6-4eba-83e3-d3ca24d8e69d"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-4555b31f-ddf6-4b71-9f3d-84f834785bc1\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MedInc</th>\n",
       "      <th>HouseAge</th>\n",
       "      <th>AveRooms</th>\n",
       "      <th>AveBedrms</th>\n",
       "      <th>Population</th>\n",
       "      <th>AveOccup</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>MedianHouseValue</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.6875</td>\n",
       "      <td>44.0</td>\n",
       "      <td>4.524476</td>\n",
       "      <td>0.993007</td>\n",
       "      <td>457.0</td>\n",
       "      <td>3.195804</td>\n",
       "      <td>34.04</td>\n",
       "      <td>-118.15</td>\n",
       "      <td>1.625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3.9917</td>\n",
       "      <td>25.0</td>\n",
       "      <td>5.576364</td>\n",
       "      <td>1.034545</td>\n",
       "      <td>1602.0</td>\n",
       "      <td>2.912727</td>\n",
       "      <td>38.33</td>\n",
       "      <td>-122.82</td>\n",
       "      <td>2.441</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.6900</td>\n",
       "      <td>18.0</td>\n",
       "      <td>4.423529</td>\n",
       "      <td>1.145882</td>\n",
       "      <td>994.0</td>\n",
       "      <td>2.338824</td>\n",
       "      <td>34.14</td>\n",
       "      <td>-116.32</td>\n",
       "      <td>0.542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.9643</td>\n",
       "      <td>52.0</td>\n",
       "      <td>4.797980</td>\n",
       "      <td>1.020202</td>\n",
       "      <td>467.0</td>\n",
       "      <td>2.358586</td>\n",
       "      <td>37.84</td>\n",
       "      <td>-122.26</td>\n",
       "      <td>1.888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6.6704</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.829317</td>\n",
       "      <td>1.022088</td>\n",
       "      <td>1503.0</td>\n",
       "      <td>3.018072</td>\n",
       "      <td>33.88</td>\n",
       "      <td>-117.99</td>\n",
       "      <td>2.406</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-4555b31f-ddf6-4b71-9f3d-84f834785bc1')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-4555b31f-ddf6-4b71-9f3d-84f834785bc1 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-4555b31f-ddf6-4b71-9f3d-84f834785bc1');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "   MedInc  HouseAge  AveRooms  AveBedrms  Population  AveOccup  Latitude  \\\n",
       "0  3.6875      44.0  4.524476   0.993007       457.0  3.195804     34.04   \n",
       "1  3.9917      25.0  5.576364   1.034545      1602.0  2.912727     38.33   \n",
       "2  1.6900      18.0  4.423529   1.145882       994.0  2.338824     34.14   \n",
       "3  3.9643      52.0  4.797980   1.020202       467.0  2.358586     37.84   \n",
       "4  6.6704      25.0  6.829317   1.022088      1503.0  3.018072     33.88   \n",
       "\n",
       "   Longitude  MedianHouseValue  \n",
       "0    -118.15             1.625  \n",
       "1    -122.82             2.441  \n",
       "2    -116.32             0.542  \n",
       "3    -122.26             1.888  \n",
       "4    -117.99             2.406  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "pd.read_csv(train_filepaths[1]).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c045a053",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "c045a053",
    "outputId": "63f1c3f2-ac16-455e-e4bf-84142bb4fb43"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['datasets/housing/my_train_00.csv',\n",
       " 'datasets/housing/my_train_01.csv',\n",
       " 'datasets/housing/my_train_02.csv',\n",
       " 'datasets/housing/my_train_03.csv',\n",
       " 'datasets/housing/my_train_04.csv',\n",
       " 'datasets/housing/my_train_05.csv',\n",
       " 'datasets/housing/my_train_06.csv',\n",
       " 'datasets/housing/my_train_07.csv',\n",
       " 'datasets/housing/my_train_08.csv',\n",
       " 'datasets/housing/my_train_09.csv',\n",
       " 'datasets/housing/my_train_10.csv',\n",
       " 'datasets/housing/my_train_11.csv',\n",
       " 'datasets/housing/my_train_12.csv',\n",
       " 'datasets/housing/my_train_13.csv',\n",
       " 'datasets/housing/my_train_14.csv',\n",
       " 'datasets/housing/my_train_15.csv',\n",
       " 'datasets/housing/my_train_16.csv',\n",
       " 'datasets/housing/my_train_17.csv',\n",
       " 'datasets/housing/my_train_18.csv',\n",
       " 'datasets/housing/my_train_19.csv']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_filepaths"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "553bbfe9",
   "metadata": {
    "id": "553bbfe9"
   },
   "source": [
    "Now let's build an input pipeline and add basic preprocessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "630cafac",
   "metadata": {
    "id": "630cafac"
   },
   "outputs": [],
   "source": [
    "filepath_dataset = tf.data.Dataset.list_files(train_filepaths, seed=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9de3eb89",
   "metadata": {
    "id": "9de3eb89"
   },
   "source": [
    "Now we will use `interleave()` method to read files. At a time 5 files will be read and the first row(header) will be skipped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "814e3932",
   "metadata": {
    "id": "814e3932"
   },
   "outputs": [],
   "source": [
    "n_readers = 5\n",
    "dataset = filepath_dataset.interleave(\n",
    "    lambda filepath: tf.data.TextLineDataset(filepath).skip(1),\n",
    "    cycle_length=n_readers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "6e53f13c",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "6e53f13c",
    "outputId": "6cbad8df-3308-443d-e612-4097bb50b13b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'4.7361,7.0,7.464968152866242,1.1178343949044587,846.0,2.694267515923567,34.49,-117.27,1.745'\n",
      "b'3.6641,17.0,5.577142857142857,1.1542857142857144,511.0,2.92,40.85,-121.07,0.808'\n",
      "b'4.5909,16.0,5.475877192982456,1.0964912280701755,1357.0,2.9758771929824563,33.63,-117.71,2.418'\n",
      "b'3.6875,44.0,4.524475524475524,0.993006993006993,457.0,3.195804195804196,34.04,-118.15,1.625'\n",
      "b'2.3,25.0,5.828178694158075,0.9587628865979382,909.0,3.1237113402061856,36.25,-119.4,1.328'\n"
     ]
    }
   ],
   "source": [
    "for line in dataset.take(5):\n",
    "    print(line.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ced2028",
   "metadata": {
    "id": "0ced2028"
   },
   "source": [
    "* For interleaving to work best, it is preferable to have files of identical length; otherwise the ends of the longest files will not be interleaved. - [Géron, A. (2019)](https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a60971b",
   "metadata": {
    "id": "1a60971b"
   },
   "source": [
    "* To use parallelism; `interleave()` does not use parallelism by default and reads one line at a time from each file. However, we can use parallelism by setting `num_parallel_calls` arguments to the number of threads we want or we can set it to `tf.data.experimental.AUTOTUNE `(in this case, Tensorflow choose the right number of threads dynamically based on the available CPU). - [Géron, A. (2019)](https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc790256",
   "metadata": {
    "id": "fc790256"
   },
   "source": [
    "Now let's define a function that we will use for basic preprocessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "87c3374e",
   "metadata": {
    "id": "87c3374e"
   },
   "outputs": [],
   "source": [
    "n_inputs = 8 # X_train.shape[-1]\n",
    "\n",
    "@tf.function\n",
    "def preprocess(line):\n",
    "    defs = [0.] * n_inputs + [tf.constant([], dtype=tf.float32)]\n",
    "    fields = tf.io.decode_csv(line, record_defaults=defs)\n",
    "    x = tf.stack(fields[:-1])\n",
    "    y = tf.stack(fields[-1:])\n",
    "    return (x - X_mean) / X_std, y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "946c3b9f",
   "metadata": {
    "id": "946c3b9f"
   },
   "source": [
    "Now it is time to combine all the stuff inside one function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "43834b72",
   "metadata": {
    "id": "43834b72"
   },
   "outputs": [],
   "source": [
    "def csv_reader_dataset(filepaths, repeat=1, n_readers=5,\n",
    "                       n_read_threads=None, shuffle_buffer_size=10000,\n",
    "                       n_parse_threads=5, batch_size=32):\n",
    "    dataset = tf.data.Dataset.list_files(filepaths).repeat(repeat)\n",
    "    dataset = dataset.interleave(\n",
    "        lambda filepath: tf.data.TextLineDataset(filepath).skip(1),\n",
    "        cycle_length=n_readers, num_parallel_calls=n_read_threads)\n",
    "    dataset = dataset.shuffle(shuffle_buffer_size)\n",
    "    dataset = dataset.map(preprocess, num_parallel_calls=n_parse_threads)\n",
    "    dataset = dataset.batch(batch_size)\n",
    "    return dataset.prefetch(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffdaf100",
   "metadata": {
    "id": "ffdaf100"
   },
   "source": [
    "At the end, we call `prefect(1)`, by doing that we create a dataset that will always be one batch ahead. In other words, while the algorithm is training, the dataset will be working in parallel to get the next batch ready. Doing that can improve the performance dramatically.\n",
    "\n",
    "Let's use the function on training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "8566d72a",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8566d72a",
    "outputId": "aa623bec-b802-4cf8-f0de-aa4dbc2d61e5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X = tf.Tensor(\n",
      "[[ 0.5804519  -0.20762321  0.05616303 -0.15191229  0.01343246  0.00604472\n",
      "   1.2525111  -1.3671792 ]\n",
      " [ 5.818099    1.8491895   1.1784915   0.28173092 -1.2496178  -0.3571987\n",
      "   0.7231292  -1.0023477 ]\n",
      " [-0.9253566   0.5834586  -0.7807257  -0.28213993 -0.36530012  0.27389365\n",
      "  -0.76194876  0.72684526]], shape=(3, 8), dtype=float32)\n",
      "y = tf.Tensor(\n",
      "[[1.752]\n",
      " [1.313]\n",
      " [1.535]], shape=(3, 1), dtype=float32)\n",
      "\n",
      "X = tf.Tensor(\n",
      "[[-0.8324941   0.6625668  -0.20741376 -0.18699841 -0.14536144  0.09635526\n",
      "   0.9807942  -0.67250353]\n",
      " [-0.62183803  0.5834586  -0.19862501 -0.3500319  -1.1437552  -0.3363751\n",
      "   1.107282   -0.8674123 ]\n",
      " [ 0.8683102   0.02970133  0.3427381  -0.29872298  0.7124906   0.28026953\n",
      "  -0.72915536  0.86178064]], shape=(3, 8), dtype=float32)\n",
      "y = tf.Tensor(\n",
      "[[0.919]\n",
      " [1.028]\n",
      " [2.182]], shape=(3, 1), dtype=float32)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "train_set = csv_reader_dataset(train_filepaths, batch_size=3)\n",
    "for X_batch, y_batch in train_set.take(2):\n",
    "    print(\"X =\", X_batch)\n",
    "    print(\"y =\", y_batch)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a123b950",
   "metadata": {
    "id": "a123b950"
   },
   "source": [
    "Let's use the function on all the sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "2047fdc2",
   "metadata": {
    "id": "2047fdc2"
   },
   "outputs": [],
   "source": [
    "train_set = csv_reader_dataset(train_filepaths, repeat=None)\n",
    "valid_set = csv_reader_dataset(valid_filepaths)\n",
    "test_set = csv_reader_dataset(test_filepaths)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c44994d",
   "metadata": {
    "id": "2c44994d"
   },
   "source": [
    "Let's train a simple model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "5484461c",
   "metadata": {
    "id": "5484461c"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"relu\", input_shape=X_train.shape[1:]),\n",
    "    keras.layers.Dense(1),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "a7c3fe27",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "a7c3fe27",
    "outputId": "21487ed1-d89e-497e-9bcd-29b90191bbe4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/keras/optimizers/optimizer_v2/gradient_descent.py:108: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  super(SGD, self).__init__(name, **kwargs)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "362/362 [==============================] - 3s 9ms/step - loss: 1.4679 - val_loss: 21.5124\n",
      "Epoch 2/10\n",
      "362/362 [==============================] - 3s 7ms/step - loss: 0.8735 - val_loss: 0.6648\n",
      "Epoch 3/10\n",
      "362/362 [==============================] - 2s 6ms/step - loss: 0.6317 - val_loss: 0.6196\n",
      "Epoch 4/10\n",
      "362/362 [==============================] - 2s 5ms/step - loss: 0.5933 - val_loss: 0.5669\n",
      "Epoch 5/10\n",
      "362/362 [==============================] - 2s 4ms/step - loss: 0.5629 - val_loss: 0.5402\n",
      "Epoch 6/10\n",
      "362/362 [==============================] - 1s 4ms/step - loss: 0.5693 - val_loss: 0.5209\n",
      "Epoch 7/10\n",
      "362/362 [==============================] - 1s 4ms/step - loss: 0.5231 - val_loss: 0.6130\n",
      "Epoch 8/10\n",
      "362/362 [==============================] - 1s 3ms/step - loss: 0.5074 - val_loss: 0.4818\n",
      "Epoch 9/10\n",
      "362/362 [==============================] - 1s 4ms/step - loss: 0.4963 - val_loss: 0.4904\n",
      "Epoch 10/10\n",
      "362/362 [==============================] - 2s 5ms/step - loss: 0.5023 - val_loss: 0.4585\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fe015779e10>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss=\"mse\", optimizer=keras.optimizers.SGD(lr=1e-3))\n",
    "\n",
    "batch_size = 32\n",
    "model.fit(train_set, steps_per_epoch=len(X_train) // batch_size, epochs=10,\n",
    "          validation_data=valid_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "20944410",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "20944410",
    "outputId": "5a61ba59-588f-4ef3-da75-9bf560b4326b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "161/161 [==============================] - 1s 4ms/step - loss: 0.4788\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.4787752032279968"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(test_set, steps=len(X_test) // batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "b8af034d",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "b8af034d",
    "outputId": "843aa458-8c87-4fe5-ccbb-ef004aff41bb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "161/161 [==============================] - 1s 3ms/step\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[3.8256302],\n",
       "       [2.41031  ],\n",
       "       [1.0489031],\n",
       "       ...,\n",
       "       [3.1952968],\n",
       "       [1.4562888],\n",
       "       [3.159451 ]], dtype=float32)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_set = test_set.map(lambda X, y: X) # we could instead just pass test_set, Keras would ignore the labels\n",
    "X_new = X_test\n",
    "model.predict(new_set, steps=len(X_new) // batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8685964",
   "metadata": {
    "id": "a8685964"
   },
   "source": [
    "As we did previously, we can also build our own training loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "60443a20",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "60443a20",
    "outputId": "9312557b-5ffe-481d-cc4c-3f04f55c4538"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/keras/optimizers/optimizer_v2/nadam.py:78: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  super(Nadam, self).__init__(name, **kwargs)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global step 100 / 1810\n",
      "Global step 200 / 1810\n",
      "Global step 300 / 1810\n",
      "Global step 400 / 1810\n",
      "Global step 500 / 1810\n",
      "Global step 600 / 1810\n",
      "Global step 700 / 1810\n",
      "Global step 800 / 1810\n",
      "Global step 900 / 1810\n",
      "Global step 1000 / 1810\n",
      "Global step 1100 / 1810\n",
      "Global step 1200 / 1810\n",
      "Global step 1300 / 1810\n",
      "Global step 1400 / 1810\n",
      "Global step 1500 / 1810\n",
      "Global step 1600 / 1810\n",
      "Global step 1700 / 1810\n",
      "Global step 1800 / 1810\n"
     ]
    }
   ],
   "source": [
    "optimizer = keras.optimizers.Nadam(lr=0.01)\n",
    "loss_fn = keras.losses.mean_squared_error\n",
    "\n",
    "@tf.function\n",
    "def train(model, n_epochs, batch_size=32,\n",
    "          n_readers=5, n_read_threads=5, shuffle_buffer_size=10000, n_parse_threads=5):\n",
    "    train_set = csv_reader_dataset(train_filepaths, repeat=n_epochs, n_readers=n_readers,\n",
    "                       n_read_threads=n_read_threads, shuffle_buffer_size=shuffle_buffer_size,\n",
    "                       n_parse_threads=n_parse_threads, batch_size=batch_size)\n",
    "    n_steps_per_epoch = len(X_train) // batch_size\n",
    "    total_steps = n_epochs * n_steps_per_epoch\n",
    "    global_step = 0\n",
    "    for X_batch, y_batch in train_set.take(total_steps):\n",
    "        global_step += 1\n",
    "        if tf.equal(global_step % 100, 0):\n",
    "            tf.print(\"\\rGlobal step\", global_step, \"/\", total_steps)\n",
    "        with tf.GradientTape() as tape:\n",
    "            y_pred = model(X_batch)\n",
    "            main_loss = tf.reduce_mean(loss_fn(y_batch, y_pred))\n",
    "            loss = tf.add_n([main_loss] + model.losses)\n",
    "        gradients = tape.gradient(loss, model.trainable_variables)\n",
    "        optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n",
    "\n",
    "train(model, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f452f64",
   "metadata": {
    "id": "9f452f64"
   },
   "source": [
    "Lastly, there is a short description of each method in the `Dataset` class which we can easily access:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "f003a18c",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "f003a18c",
    "outputId": "96fd8afe-46b5-47aa-a255-168d832a238c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "● apply()              Applies a transformation function to this dataset.\n",
      "● as_numpy_iterator()  Returns an iterator which converts all elements of the dataset to numpy.\n",
      "● batch()              Combines consecutive elements of this dataset into batches.\n",
      "● bucket_by_sequence_length()A transformation that buckets elements in a `Dataset` by length.\n",
      "● cache()              Caches the elements in this dataset.\n",
      "● cardinality()        Returns the cardinality of the dataset, if known.\n",
      "● choose_from_datasets()Creates a dataset that deterministically chooses elements from `datasets`.\n",
      "● concatenate()        Creates a `Dataset` by concatenating the given dataset with this dataset.\n",
      "● element_spec()       The type specification of an element of this dataset.\n",
      "● enumerate()          Enumerates the elements of this dataset.\n",
      "● filter()             Filters this dataset according to `predicate`.\n",
      "● flat_map()           Maps `map_func` across this dataset and flattens the result.\n",
      "● from_generator()     Creates a `Dataset` whose elements are generated by `generator`. (deprecated arguments)\n",
      "● from_tensor_slices() Creates a `Dataset` whose elements are slices of the given tensors.\n",
      "● from_tensors()       Creates a `Dataset` with a single element, comprising the given tensors.\n",
      "● get_single_element() Returns the single element of the `dataset`.\n",
      "● group_by_window()    Groups windows of elements by key and reduces them.\n",
      "● interleave()         Maps `map_func` across this dataset, and interleaves the results.\n",
      "● list_files()         A dataset of all files matching one or more glob patterns.\n",
      "● map()                Maps `map_func` across the elements of this dataset.\n",
      "● options()            Returns the options for this dataset and its inputs.\n",
      "● padded_batch()       Combines consecutive elements of this dataset into padded batches.\n",
      "● prefetch()           Creates a `Dataset` that prefetches elements from this dataset.\n",
      "● random()             Creates a `Dataset` of pseudorandom values.\n",
      "● range()              Creates a `Dataset` of a step-separated range of values.\n",
      "● reduce()             Reduces the input dataset to a single element.\n",
      "● rejection_resample() A transformation that resamples a dataset to a target distribution.\n",
      "● repeat()             Repeats this dataset so each original value is seen `count` times.\n",
      "● sample_from_datasets()Samples elements at random from the datasets in `datasets`.\n",
      "● scan()               A transformation that scans a function across an input dataset.\n",
      "● shard()              Creates a `Dataset` that includes only 1/`num_shards` of this dataset.\n",
      "● shuffle()            Randomly shuffles the elements of this dataset.\n",
      "● skip()               Creates a `Dataset` that skips `count` elements from this dataset.\n",
      "● snapshot()           API to persist the output of the input dataset.\n",
      "● take()               Creates a `Dataset` with at most `count` elements from this dataset.\n",
      "● take_while()         A transformation that stops dataset iteration based on a `predicate`.\n",
      "● unbatch()            Splits elements of a dataset into multiple elements.\n",
      "● unique()             A transformation that discards duplicate elements of a `Dataset`.\n",
      "● window()             Returns a dataset of \"windows\".\n",
      "● with_options()       Returns a new `tf.data.Dataset` with the given options set.\n",
      "● zip()                Creates a `Dataset` by zipping together the given datasets.\n"
     ]
    }
   ],
   "source": [
    "for m in dir(tf.data.Dataset):\n",
    "    if not (m.startswith(\"_\") or m.endswith(\"_\")):\n",
    "        func = getattr(tf.data.Dataset, m)\n",
    "        if hasattr(func, \"__doc__\"):\n",
    "            print(\"● {:21s}{}\".format(m + \"()\", func.__doc__.split(\"\\n\")[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b99b55e8",
   "metadata": {
    "id": "b99b55e8"
   },
   "source": [
    "## Preprocessing Layers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3de23484",
   "metadata": {
    "id": "3de23484"
   },
   "source": [
    "I generally prefer to do preprocessing before constructing the model but in some cases, it may be also good to use Preprocessing Layers. It's especially good (for me at least) to use them for data augmentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "caadf923",
   "metadata": {
    "id": "caadf923"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "def load_housing_data():\n",
    "    csv_path = os.path.join(\"/content/housing.csv\")\n",
    "    return pd.read_csv(csv_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "cb868861",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "cb868861",
    "outputId": "dc3e6e66-854e-4bdd-9c98-935fca076470"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-434f2182-1568-4aa8-8c0d-c55509d38f07\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "      <th>ocean_proximity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-122.23</td>\n",
       "      <td>37.88</td>\n",
       "      <td>41.0</td>\n",
       "      <td>880.0</td>\n",
       "      <td>129.0</td>\n",
       "      <td>322.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>8.3252</td>\n",
       "      <td>452600.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-122.22</td>\n",
       "      <td>37.86</td>\n",
       "      <td>21.0</td>\n",
       "      <td>7099.0</td>\n",
       "      <td>1106.0</td>\n",
       "      <td>2401.0</td>\n",
       "      <td>1138.0</td>\n",
       "      <td>8.3014</td>\n",
       "      <td>358500.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-122.24</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1467.0</td>\n",
       "      <td>190.0</td>\n",
       "      <td>496.0</td>\n",
       "      <td>177.0</td>\n",
       "      <td>7.2574</td>\n",
       "      <td>352100.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-122.25</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1274.0</td>\n",
       "      <td>235.0</td>\n",
       "      <td>558.0</td>\n",
       "      <td>219.0</td>\n",
       "      <td>5.6431</td>\n",
       "      <td>341300.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-122.25</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1627.0</td>\n",
       "      <td>280.0</td>\n",
       "      <td>565.0</td>\n",
       "      <td>259.0</td>\n",
       "      <td>3.8462</td>\n",
       "      <td>342200.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-434f2182-1568-4aa8-8c0d-c55509d38f07')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-434f2182-1568-4aa8-8c0d-c55509d38f07 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-434f2182-1568-4aa8-8c0d-c55509d38f07');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "   longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "0    -122.23     37.88                41.0        880.0           129.0   \n",
       "1    -122.22     37.86                21.0       7099.0          1106.0   \n",
       "2    -122.24     37.85                52.0       1467.0           190.0   \n",
       "3    -122.25     37.85                52.0       1274.0           235.0   \n",
       "4    -122.25     37.85                52.0       1627.0           280.0   \n",
       "\n",
       "   population  households  median_income  median_house_value ocean_proximity  \n",
       "0       322.0       126.0         8.3252            452600.0        NEAR BAY  \n",
       "1      2401.0      1138.0         8.3014            358500.0        NEAR BAY  \n",
       "2       496.0       177.0         7.2574            352100.0        NEAR BAY  \n",
       "3       558.0       219.0         5.6431            341300.0        NEAR BAY  \n",
       "4       565.0       259.0         3.8462            342200.0        NEAR BAY  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing = load_housing_data()\n",
    "housing.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34562bf9",
   "metadata": {
    "id": "34562bf9"
   },
   "source": [
    "We can perform normalization by using Tensorflow. I will outline two ways here, the first one is to use `tf.feature_column.numeric_column()` and the other one is using `keras.layers.Lambda()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "f1264e6f",
   "metadata": {
    "id": "f1264e6f"
   },
   "outputs": [],
   "source": [
    "age_mean, age_std = X_mean[1], X_std[1]  # The median age is column in 1\n",
    "housing_median_age = tf.feature_column.numeric_column(\n",
    "    \"housing_median_age\", normalizer_fn=lambda x: (x - age_mean) / age_std)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "8bd34ac6",
   "metadata": {
    "id": "8bd34ac6"
   },
   "outputs": [],
   "source": [
    "means = np.mean(X_train, axis=0, keepdims=True)\n",
    "stds = np.std(X_train, axis=0, keepdims=True)\n",
    "eps = keras.backend.epsilon()\n",
    "model = keras.models.Sequential([\n",
    " keras.layers.Lambda(lambda inputs: (inputs - means) / (stds + eps)),\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08e23db0",
   "metadata": {
    "id": "08e23db0"
   },
   "source": [
    "We can also perform one-hot encoding easily with Tensorflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "a4ae75ac",
   "metadata": {
    "id": "a4ae75ac"
   },
   "outputs": [],
   "source": [
    "vocab = [\"<1H OCEAN\", \"INLAND\", \"NEAR OCEAN\", \"NEAR BAY\", \"ISLAND\"]\n",
    "indices = tf.range(len(vocab), dtype=tf.int64)\n",
    "\n",
    "table_init = tf.lookup.KeyValueTensorInitializer(vocab, indices)\n",
    "num_oov_buckets = 2\n",
    "table = tf.lookup.StaticVocabularyTable(table_init, num_oov_buckets)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f1646e7",
   "metadata": {
    "id": "7f1646e7"
   },
   "source": [
    "The code is quite simple. \n",
    "\n",
    "1. We started by defining our vocabulary (possible categories) and then created a tensor for the instances.\n",
    "2. We then passed the list of categories and their corresponding indices to create an initializer for the lookup table. Note: `KeyValueTensorInitializer()` is used when we have the data ready, in the case when we categories are listed in a text file we should prefer `TextFileInitializer()` instead.\n",
    "3. In the last two lines we created the lookup table by specifying the number of out-of-vocabulary (OOV) buckets. \n",
    "\n",
    "**Out-of-Vocabulary Buckets:** These are the categories that does not exist in the vocabulary. When there is a category that does not exist in the vocabulary, the lookup table will compute a hash of this category and assign the unknown category to one of these out-of-vocabulary buckets. \n",
    "\n",
    "* In the case when we have a large number of categories, It may not be convenient to get the full list of categories. A good solution to that problem is to define the vocabulary based on a data sample rather than the whole training set and add some oov buckets for the other categories that were not in the data sample. The more unknown categories you expect to find during training, the more oov buckets you should use.Indeed, if there are not enough oov buckets, there will be collisions: different categories will end up in the same bucket, so the neural network will not be able to distinguish them (at least not based on this feature). - [Géron, A. (2019)](https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03f24fd1",
   "metadata": {
    "id": "03f24fd1"
   },
   "source": [
    "Let's do another example for one-hot encoding using `tf.one_hot()` to see oov buckets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "93bc327b",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "93bc327b",
    "outputId": "5c0bb3ac-4811-4cf7-c8f3-22637b983d14"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4,), dtype=int64, numpy=array([3, 5, 1, 1])>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categories = tf.constant([\"NEAR BAY\", \"DESERT\", \"INLAND\", \"INLAND\"])\n",
    "cat_indices = table.lookup(categories)\n",
    "cat_indices\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdb7f3d5",
   "metadata": {
    "id": "bdb7f3d5"
   },
   "source": [
    "The unknown category \"DESERT\" was mapped to one of the two oov buckets. Let's use `tf.one_hot()` to encode the indices now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "45088cfb",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "45088cfb",
    "outputId": "3319366f-353c-4530-df77-f76b89451ff0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4, 7), dtype=float32, numpy=\n",
       "array([[0., 0., 0., 1., 0., 0., 0.],\n",
       "       [0., 0., 0., 0., 0., 1., 0.],\n",
       "       [0., 1., 0., 0., 0., 0., 0.],\n",
       "       [0., 1., 0., 0., 0., 0., 0.]], dtype=float32)>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_one_hot = tf.one_hot(cat_indices, depth=len(vocab) + num_oov_buckets)\n",
    "cat_one_hot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2465c297",
   "metadata": {
    "id": "2465c297"
   },
   "source": [
    "* As a rule of thumb, if the number of categories is lower than 10, then one-hot encoding is generally the way to go (but your mileage may vary!). If the number of categories is greater than 50 (which is often the case when you use hash buckets), then embeddings are usually preferable. In between 10 and 50 categories, you may want to experiment with both options and see which one works best for your use case."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5247137a",
   "metadata": {
    "id": "5247137a"
   },
   "source": [
    "### Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1888b68b",
   "metadata": {
    "id": "1888b68b"
   },
   "source": [
    "Embeddings are an important subject that we will cover in Sequences Models but let's have a quick introduction about this subject since this section is about encoding.\n",
    "\n",
    "* An embedding is a trainable dense vector that represents a category. By default, embeddings are initialized randomly, so for example the \"NEAR BAY\" category could be represented initially by a random vector such as [0.131, 0.890], while the \"NEAR OCEAN\" category might be represented by another random vector such as [0.631, 0.791]. In this example, we use 2D embeddings, but the number of dimensions is a hyperparameter you can tweak. - [Géron, A. (2019)](https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/) \n",
    "\n",
    "It is an important thing that we can tweak the dimension, thanks to that, we can train the embedding and find a better representation which in return, will make it easier for our neural network to make accurate predictions. This is a subject of representation learning that we will talk more about later.\n",
    "\n",
    "**Word Embeddings:** Word Embeddings got very popular after Google Researchers publisted the [paper](https://arxiv.org/abs/1310.4546). In this paper, they traied a neural network to predict the words neay any given word, and obtained astounding word embeddings. These word embeddings were so powerful that they even captured the concept of gender (The famous example: if you compute embedding vectors of the following ford, King – Man + Woman (adding and subtracting then the result will be very close to the embedding of the word Queen). On the other hand, word embeddings can also sometimes capture out worst biases (a related [paper](https://arxiv.org/abs/1905.09866))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58f6b9a6",
   "metadata": {
    "id": "58f6b9a6"
   },
   "source": [
    "Let's implement a simple embedding.\n",
    "\n",
    "We firstly need to create an embedding matrix that has one row per category and per oov bucket, and one column per embedding\n",
    "dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "1dd94473",
   "metadata": {
    "id": "1dd94473"
   },
   "outputs": [],
   "source": [
    "embedding_dim = 2\n",
    "embed_init = tf.random.uniform([len(vocab) + num_oov_buckets, embedding_dim])\n",
    "embedding_matrix = tf.Variable(embed_init)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "835be69e",
   "metadata": {
    "id": "835be69e"
   },
   "source": [
    "In this example, we have a 2D embedding which is a random 6 x 2 matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "7281272e",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7281272e",
    "outputId": "4daafa11-d4f8-498c-a992-67c1f4b0b522"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=(7, 2) dtype=float32, numpy=\n",
       "array([[0.7413678 , 0.62854624],\n",
       "       [0.01738465, 0.3431449 ],\n",
       "       [0.51063764, 0.3777541 ],\n",
       "       [0.07321596, 0.02137029],\n",
       "       [0.2871771 , 0.4710616 ],\n",
       "       [0.6936141 , 0.07321334],\n",
       "       [0.93251204, 0.20843053]], dtype=float32)>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fe1e239",
   "metadata": {
    "id": "8fe1e239"
   },
   "source": [
    "Now let’s encode the same batch of categorical features as we did earlier, but using these embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "e2eed29d",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "e2eed29d",
    "outputId": "727073bd-4eaf-4417-803b-38ed8c04abcf"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4,), dtype=int64, numpy=array([3, 5, 1, 1])>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categories = tf.constant([\"NEAR BAY\", \"DESERT\", \"INLAND\", \"INLAND\"])\n",
    "cat_indices = table.lookup(categories)\n",
    "cat_indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "592f1673",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "592f1673",
    "outputId": "c4542f3d-6242-4ce6-d159-88bab37c4435"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4, 2), dtype=float32, numpy=\n",
       "array([[0.07321596, 0.02137029],\n",
       "       [0.6936141 , 0.07321334],\n",
       "       [0.01738465, 0.3431449 ],\n",
       "       [0.01738465, 0.3431449 ]], dtype=float32)>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.nn.embedding_lookup(embedding_matrix, cat_indices)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff81a343",
   "metadata": {
    "id": "ff81a343"
   },
   "source": [
    "The tf.nn.embedding_lookup() function looks up the rows in the embedding matrix, an returns the embedding for each row in the embedding matrix.\n",
    "\n",
    "Instead of doing what we did above, we can also use keras.layers.Embedding which handles the embedding matrix, is trainable by default and randomly initialized.When it is called with some category indices it returns the rows at those indices in the embedding matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "b3ac7398",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "b3ac7398",
    "outputId": "b3dd2950-fec4-46fb-c9a7-78bfec25dbed"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4, 2), dtype=float32, numpy=\n",
       "array([[ 0.01289039,  0.0191061 ],\n",
       "       [ 0.01639661, -0.01945841],\n",
       "       [ 0.00692506, -0.00518861],\n",
       "       [ 0.00692506, -0.00518861]], dtype=float32)>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding = keras.layers.Embedding(input_dim=len(vocab) + num_oov_buckets,output_dim=embedding_dim)\n",
    "embedding(cat_indices)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1bae7ea",
   "metadata": {
    "id": "c1bae7ea"
   },
   "source": [
    "Let's put everything together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "4af54c49",
   "metadata": {
    "id": "4af54c49"
   },
   "outputs": [],
   "source": [
    "regular_inputs = keras.layers.Input(shape=[8])\n",
    "categories = keras.layers.Input(shape=[], dtype=tf.string)\n",
    "cat_indices = keras.layers.Lambda(lambda cats: table.lookup(cats))(categories)\n",
    "cat_embed = keras.layers.Embedding(input_dim=6, output_dim=2)(cat_indices)\n",
    "encoded_inputs = keras.layers.concatenate([regular_inputs, cat_embed])\n",
    "outputs = keras.layers.Dense(1)(encoded_inputs)\n",
    "model = keras.models.Model(inputs=[regular_inputs, categories],\n",
    " outputs=[outputs])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98bbd3af",
   "metadata": {
    "id": "98bbd3af"
   },
   "source": [
    "This model gets two inputs: one for regular input that contains eight numerical features and the other one is a categorical input (containing one categorical feature per instance). It uses the Lambda layer to look up each category’s index, then it looks up the embeddings for these indices. To give the encoded input, we concatenate the embeddings and the regular inputs. An lastly, we add our dense output layer.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a54995f",
   "metadata": {
    "id": "8a54995f"
   },
   "source": [
    "* One-hot encoding followed by a Dense layer (with no activation function and no biases) is equivalent to an Embedding layer. However, the Embedding layer uses way fewer computations (the performance difference becomes clear when the size of the embedding matrix grows). The Dense layer’s weight matrix plays the role of the embedding matrix. For example, using one-hot vectors of size 20 and a Dense layer with 10 units is equivalent to using an Embedding layer with input_dim=20 and output_dim=10. As a result, it would be wasteful to use more embedding dimensions than the number of units in the layer that follows the Embedding layer. - [Géron, A. (2019)](https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "becccd6c",
   "metadata": {
    "id": "becccd6c"
   },
   "source": [
    "## Keras Preprocessing Layers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "798f4b8a",
   "metadata": {
    "id": "798f4b8a"
   },
   "source": [
    "Keras provides layers that we can use for preprocessing. The API provides layers such `keras.layers.Normalization()` or `keras.layers.TextVectorization()` that we previously used as well as `keras.layers.Discretization()` that we didn't use before.\n",
    "Discretization layer is used for one-hot encoding continuous data in bins, for instance, (low, medium, high) would be encoded as [1, 0, 0], [0, 1, 0], and [0, 0, 1]. \n",
    "\n",
    "\n",
    "Additionally, we can chain multiple preprocessing layers by using keras.layers.PreprocessingStage(). The TextVectorization layer can also be used to output word-count vectors instead of word indices (a.k.a bag of words: named so because in this representation the order of the words are not important, for instance, if the vocabulary is [\"and\", \"basketball\", \"more\"] then the text \"more and more\" will be mapped to the vector [1, 0, 2] since more appeared for twice and and appeared once). We can also use TF-IDF (A common technique that is used to divide each word count by the log of the total number of training instances in which the word appears. The reason for doing that is to reduce the importance of frequent words so that words like \"and\" that will probably appear frequently in most text even though they are the least interesting, will not perceived unnecessarly important by the algorithm.\n",
    "\n",
    "It is important to note that these layers are not differentiable and will be freezed during training. More information can be found on [Github](https://github.com/keras-team/governance/blob/master/rfcs/20190502-preprocessing-layers.md)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd0e2eb7",
   "metadata": {
    "id": "bd0e2eb7"
   },
   "source": [
    "Tensorflow has many layers that we can use for a wide number of operations. See the [documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d78a006",
   "metadata": {
    "id": "4d78a006"
   },
   "source": [
    "## Tensorflow Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4825aaf",
   "metadata": {
    "id": "f4825aaf"
   },
   "source": [
    "We can easily load datasets that are already existing in Tensorflow. Let's look the list of datasets that are avaliable in Tensorflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "6d866793",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "6d866793",
    "outputId": "70dc6f8c-d902-4d7b-f078-5ab42e80f38e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['abstract_reasoning', 'accentdb', 'aeslc', 'aflw2k3d', 'ag_news_subset', 'ai2_arc', 'ai2_arc_with_ir', 'amazon_us_reviews', 'anli', 'answer_equivalence', 'arc', 'asqa', 'asset', 'assin2', 'bair_robot_pushing_small', 'bccd', 'beans', 'bee_dataset', 'beir', 'big_patent', 'bigearthnet', 'billsum', 'binarized_mnist', 'binary_alpha_digits', 'ble_wind_field', 'blimp', 'booksum', 'bool_q', 'c4', 'caltech101', 'caltech_birds2010', 'caltech_birds2011', 'cardiotox', 'cars196', 'cassava', 'cats_vs_dogs', 'celeb_a', 'celeb_a_hq', 'cfq', 'cherry_blossoms', 'chexpert', 'cifar10', 'cifar100', 'cifar10_1', 'cifar10_corrupted', 'citrus_leaves', 'cityscapes', 'civil_comments', 'clevr', 'clic', 'clinc_oos', 'cmaterdb', 'cnn_dailymail', 'coco', 'coco_captions', 'coil100', 'colorectal_histology', 'colorectal_histology_large', 'common_voice', 'coqa', 'cos_e', 'cosmos_qa', 'covid19', 'covid19sum', 'crema_d', 'criteo', 'cs_restaurants', 'curated_breast_imaging_ddsm', 'cycle_gan', 'd4rl_adroit_door', 'd4rl_adroit_hammer', 'd4rl_adroit_pen', 'd4rl_adroit_relocate', 'd4rl_antmaze', 'd4rl_mujoco_ant', 'd4rl_mujoco_halfcheetah', 'd4rl_mujoco_hopper', 'd4rl_mujoco_walker2d', 'dart', 'davis', 'deep1b', 'deep_weeds', 'definite_pronoun_resolution', 'dementiabank', 'diabetic_retinopathy_detection', 'diamonds', 'div2k', 'dmlab', 'doc_nli', 'dolphin_number_word', 'domainnet', 'downsampled_imagenet', 'drop', 'dsprites', 'dtd', 'duke_ultrasound', 'e2e_cleaned', 'efron_morris75', 'emnist', 'eraser_multi_rc', 'esnli', 'eurosat', 'fashion_mnist', 'flic', 'flores', 'food101', 'forest_fires', 'fuss', 'gap', 'geirhos_conflict_stimuli', 'gem', 'genomics_ood', 'german_credit_numeric', 'gigaword', 'glove100_angular', 'glue', 'goemotions', 'gov_report', 'gpt3', 'gref', 'groove', 'grounded_scan', 'gsm8k', 'gtzan', 'gtzan_music_speech', 'hellaswag', 'higgs', 'hillstrom', 'horses_or_humans', 'howell', 'i_naturalist2017', 'i_naturalist2018', 'imagenet2012', 'imagenet2012_corrupted', 'imagenet2012_fewshot', 'imagenet2012_multilabel', 'imagenet2012_real', 'imagenet2012_subset', 'imagenet_a', 'imagenet_lt', 'imagenet_r', 'imagenet_resized', 'imagenet_sketch', 'imagenet_v2', 'imagenette', 'imagewang', 'imdb_reviews', 'irc_disentanglement', 'iris', 'istella', 'kddcup99', 'kitti', 'kmnist', 'lambada', 'lfw', 'librispeech', 'librispeech_lm', 'libritts', 'ljspeech', 'lm1b', 'locomotion', 'lost_and_found', 'lsun', 'lvis', 'malaria', 'math_dataset', 'math_qa', 'mctaco', 'media_sum', 'mlqa', 'mnist', 'mnist_corrupted', 'movie_lens', 'movie_rationales', 'movielens', 'moving_mnist', 'mrqa', 'mslr_web', 'mt_opt', 'multi_news', 'multi_nli', 'multi_nli_mismatch', 'natural_questions', 'natural_questions_open', 'newsroom', 'nsynth', 'nyu_depth_v2', 'ogbg_molpcba', 'omniglot', 'open_images_challenge2019_detection', 'open_images_v4', 'openbookqa', 'opinion_abstracts', 'opinosis', 'opus', 'oxford_flowers102', 'oxford_iiit_pet', 'para_crawl', 'pass', 'patch_camelyon', 'paws_wiki', 'paws_x_wiki', 'penguins', 'pet_finder', 'pg19', 'piqa', 'places365_small', 'plant_leaves', 'plant_village', 'plantae_k', 'protein_net', 'qa4mre', 'qasc', 'quac', 'quality', 'quickdraw_bitmap', 'race', 'radon', 'reddit', 'reddit_disentanglement', 'reddit_tifu', 'ref_coco', 'resisc45', 'rlu_atari', 'rlu_atari_checkpoints', 'rlu_atari_checkpoints_ordered', 'rlu_control_suite', 'rlu_dmlab_explore_object_rewards_few', 'rlu_dmlab_explore_object_rewards_many', 'rlu_dmlab_rooms_select_nonmatching_object', 'rlu_dmlab_rooms_watermaze', 'rlu_dmlab_seekavoid_arena01', 'rlu_locomotion', 'rlu_rwrl', 'robomimic_ph', 'robonet', 'robosuite_panda_pick_place_can', 'rock_paper_scissors', 'rock_you', 's3o4d', 'salient_span_wikipedia', 'samsum', 'savee', 'scan', 'scene_parse150', 'schema_guided_dialogue', 'sci_tail', 'scicite', 'scientific_papers', 'scrolls', 'sentiment140', 'shapes3d', 'sift1m', 'simpte', 'siscore', 'smallnorb', 'smartwatch_gestures', 'snli', 'so2sat', 'speech_commands', 'spoken_digit', 'squad', 'squad_question_generation', 'stanford_dogs', 'stanford_online_products', 'star_cfq', 'starcraft_video', 'stl10', 'story_cloze', 'summscreen', 'sun397', 'super_glue', 'svhn_cropped', 'symmetric_solids', 'tao', 'ted_hrlr_translate', 'ted_multi_translate', 'tedlium', 'tf_flowers', 'the300w_lp', 'tiny_shakespeare', 'titanic', 'trec', 'trivia_qa', 'tydi_qa', 'uc_merced', 'ucf101', 'unified_qa', 'vctk', 'visual_domain_decathlon', 'voc', 'voxceleb', 'voxforge', 'waymo_open_dataset', 'web_graph', 'web_nlg', 'web_questions', 'wider_face', 'wiki40b', 'wiki_auto', 'wiki_bio', 'wiki_dialog', 'wiki_table_questions', 'wiki_table_text', 'wikiann', 'wikihow', 'wikipedia', 'wikipedia_toxicity_subtypes', 'wine_quality', 'winogrande', 'wit', 'wit_kaggle', 'wmt13_translate', 'wmt14_translate', 'wmt15_translate', 'wmt16_translate', 'wmt17_translate', 'wmt18_translate', 'wmt19_translate', 'wmt_t2t_translate', 'wmt_translate', 'wordnet', 'wsc273', 'xnli', 'xquad', 'xsum', 'xtreme_pawsx', 'xtreme_s', 'xtreme_xnli', 'yelp_polarity_reviews', 'yes_no', 'youtube_vis', 'huggingface:acronym_identification', 'huggingface:ade_corpus_v2', 'huggingface:adv_glue', 'huggingface:adversarial_qa', 'huggingface:aeslc', 'huggingface:afrikaans_ner_corpus', 'huggingface:ag_news', 'huggingface:ai2_arc', 'huggingface:air_dialogue', 'huggingface:ajgt_twitter_ar', 'huggingface:allegro_reviews', 'huggingface:allocine', 'huggingface:alt', 'huggingface:amazon_polarity', 'huggingface:amazon_reviews_multi', 'huggingface:amazon_us_reviews', 'huggingface:ambig_qa', 'huggingface:americas_nli', 'huggingface:ami', 'huggingface:amttl', 'huggingface:anli', 'huggingface:app_reviews', 'huggingface:aqua_rat', 'huggingface:aquamuse', 'huggingface:ar_cov19', 'huggingface:ar_res_reviews', 'huggingface:ar_sarcasm', 'huggingface:arabic_billion_words', 'huggingface:arabic_pos_dialect', 'huggingface:arabic_speech_corpus', 'huggingface:arcd', 'huggingface:arsentd_lev', 'huggingface:art', 'huggingface:arxiv_dataset', 'huggingface:ascent_kb', 'huggingface:aslg_pc12', 'huggingface:asnq', 'huggingface:asset', 'huggingface:assin', 'huggingface:assin2', 'huggingface:atomic', 'huggingface:autshumato', 'huggingface:babi_qa', 'huggingface:banking77', 'huggingface:bbaw_egyptian', 'huggingface:bbc_hindi_nli', 'huggingface:bc2gm_corpus', 'huggingface:beans', 'huggingface:best2009', 'huggingface:bianet', 'huggingface:bible_para', 'huggingface:big_patent', 'huggingface:bigbench', 'huggingface:billsum', 'huggingface:bing_coronavirus_query_set', 'huggingface:biomrc', 'huggingface:biosses', 'huggingface:biwi_kinect_head_pose', 'huggingface:blbooks', 'huggingface:blbooksgenre', 'huggingface:blended_skill_talk', 'huggingface:blimp', 'huggingface:blog_authorship_corpus', 'huggingface:bn_hate_speech', 'huggingface:bnl_newspapers', 'huggingface:bookcorpus', 'huggingface:bookcorpusopen', 'huggingface:boolq', 'huggingface:bprec', 'huggingface:break_data', 'huggingface:brwac', 'huggingface:bsd_ja_en', 'huggingface:bswac', 'huggingface:c3', 'huggingface:c4', 'huggingface:cail2018', 'huggingface:caner', 'huggingface:capes', 'huggingface:casino', 'huggingface:catalonia_independence', 'huggingface:cats_vs_dogs', 'huggingface:cawac', 'huggingface:cbt', 'huggingface:cc100', 'huggingface:cc_news', 'huggingface:ccaligned_multilingual', 'huggingface:cdsc', 'huggingface:cdt', 'huggingface:cedr', 'huggingface:cfq', 'huggingface:chr_en', 'huggingface:cifar10', 'huggingface:cifar100', 'huggingface:circa', 'huggingface:civil_comments', 'huggingface:clickbait_news_bg', 'huggingface:climate_fever', 'huggingface:clinc_oos', 'huggingface:clue', 'huggingface:cmrc2018', 'huggingface:cmu_hinglish_dog', 'huggingface:cnn_dailymail', 'huggingface:coached_conv_pref', 'huggingface:coarse_discourse', 'huggingface:codah', 'huggingface:code_search_net', 'huggingface:code_x_glue_cc_clone_detection_big_clone_bench', 'huggingface:code_x_glue_cc_clone_detection_poj104', 'huggingface:code_x_glue_cc_cloze_testing_all', 'huggingface:code_x_glue_cc_cloze_testing_maxmin', 'huggingface:code_x_glue_cc_code_completion_line', 'huggingface:code_x_glue_cc_code_completion_token', 'huggingface:code_x_glue_cc_code_refinement', 'huggingface:code_x_glue_cc_code_to_code_trans', 'huggingface:code_x_glue_cc_defect_detection', 'huggingface:code_x_glue_ct_code_to_text', 'huggingface:code_x_glue_tc_nl_code_search_adv', 'huggingface:code_x_glue_tc_text_to_code', 'huggingface:code_x_glue_tt_text_to_text', 'huggingface:com_qa', 'huggingface:common_gen', 'huggingface:common_language', 'huggingface:common_voice', 'huggingface:commonsense_qa', 'huggingface:competition_math', 'huggingface:compguesswhat', 'huggingface:conceptnet5', 'huggingface:conceptual_12m', 'huggingface:conceptual_captions', 'huggingface:conll2000', 'huggingface:conll2002', 'huggingface:conll2003', 'huggingface:conll2012_ontonotesv5', 'huggingface:conllpp', 'huggingface:consumer-finance-complaints', 'huggingface:conv_ai', 'huggingface:conv_ai_2', 'huggingface:conv_ai_3', 'huggingface:conv_questions', 'huggingface:coqa', 'huggingface:cord19', 'huggingface:cornell_movie_dialog', 'huggingface:cos_e', 'huggingface:cosmos_qa', 'huggingface:counter', 'huggingface:covid_qa_castorini', 'huggingface:covid_qa_deepset', 'huggingface:covid_qa_ucsd', 'huggingface:covid_tweets_japanese', 'huggingface:covost2', 'huggingface:cppe-5', 'huggingface:craigslist_bargains', 'huggingface:crawl_domain', 'huggingface:crd3', 'huggingface:crime_and_punish', 'huggingface:crows_pairs', 'huggingface:cryptonite', 'huggingface:cs_restaurants', 'huggingface:cuad', 'huggingface:curiosity_dialogs', 'huggingface:daily_dialog', 'huggingface:dane', 'huggingface:danish_political_comments', 'huggingface:dart', 'huggingface:datacommons_factcheck', 'huggingface:dbpedia_14', 'huggingface:dbrd', 'huggingface:deal_or_no_dialog', 'huggingface:definite_pronoun_resolution', 'huggingface:dengue_filipino', 'huggingface:dialog_re', 'huggingface:diplomacy_detection', 'huggingface:disaster_response_messages', 'huggingface:discofuse', 'huggingface:discovery', 'huggingface:disfl_qa', 'huggingface:doc2dial', 'huggingface:docred', 'huggingface:doqa', 'huggingface:dream', 'huggingface:drop', 'huggingface:duorc', 'huggingface:dutch_social', 'huggingface:dyk', 'huggingface:e2e_nlg', 'huggingface:e2e_nlg_cleaned', 'huggingface:ecb', 'huggingface:ecthr_cases', 'huggingface:eduge', 'huggingface:ehealth_kd', 'huggingface:eitb_parcc', 'huggingface:electricity_load_diagrams', 'huggingface:eli5', 'huggingface:eli5_category', 'huggingface:elkarhizketak', 'huggingface:emea', 'huggingface:emo', 'huggingface:emotion', 'huggingface:emotone_ar', 'huggingface:empathetic_dialogues', 'huggingface:enriched_web_nlg', 'huggingface:enwik8', 'huggingface:eraser_multi_rc', 'huggingface:esnli', 'huggingface:eth_py150_open', 'huggingface:ethos', 'huggingface:ett', 'huggingface:eu_regulatory_ir', 'huggingface:eurlex', 'huggingface:euronews', 'huggingface:europa_eac_tm', 'huggingface:europa_ecdc_tm', 'huggingface:europarl_bilingual', 'huggingface:event2Mind', 'huggingface:evidence_infer_treatment', 'huggingface:exams', 'huggingface:factckbr', 'huggingface:fake_news_english', 'huggingface:fake_news_filipino', 'huggingface:farsi_news', 'huggingface:fashion_mnist', 'huggingface:fever', 'huggingface:few_rel', 'huggingface:financial_phrasebank', 'huggingface:finer', 'huggingface:flores', 'huggingface:flue', 'huggingface:food101', 'huggingface:fquad', 'huggingface:freebase_qa', 'huggingface:gap', 'huggingface:gem', 'huggingface:generated_reviews_enth', 'huggingface:generics_kb', 'huggingface:german_legal_entity_recognition', 'huggingface:germaner', 'huggingface:germeval_14', 'huggingface:giga_fren', 'huggingface:gigaword', 'huggingface:glucose', 'huggingface:glue', 'huggingface:gnad10', 'huggingface:go_emotions', 'huggingface:gooaq', 'huggingface:google_wellformed_query', 'huggingface:grail_qa', 'huggingface:great_code', 'huggingface:greek_legal_code', 'huggingface:gsm8k', 'huggingface:guardian_authorship', 'huggingface:gutenberg_time', 'huggingface:hans', 'huggingface:hansards', 'huggingface:hard', 'huggingface:harem', 'huggingface:has_part', 'huggingface:hate_offensive', 'huggingface:hate_speech18', 'huggingface:hate_speech_filipino', 'huggingface:hate_speech_offensive', 'huggingface:hate_speech_pl', 'huggingface:hate_speech_portuguese', 'huggingface:hatexplain', 'huggingface:hausa_voa_ner', 'huggingface:hausa_voa_topics', 'huggingface:hda_nli_hindi', 'huggingface:head_qa', 'huggingface:health_fact', 'huggingface:hebrew_projectbenyehuda', 'huggingface:hebrew_sentiment', 'huggingface:hebrew_this_world', 'huggingface:hellaswag', 'huggingface:hendrycks_test', 'huggingface:hind_encorp', 'huggingface:hindi_discourse', 'huggingface:hippocorpus', 'huggingface:hkcancor', 'huggingface:hlgd', 'huggingface:hope_edi', 'huggingface:hotpot_qa', 'huggingface:hover', 'huggingface:hrenwac_para', 'huggingface:hrwac', 'huggingface:humicroedit', 'huggingface:hybrid_qa', 'huggingface:hyperpartisan_news_detection', 'huggingface:iapp_wiki_qa_squad', 'huggingface:id_clickbait', 'huggingface:id_liputan6', 'huggingface:id_nergrit_corpus', 'huggingface:id_newspapers_2018', 'huggingface:id_panl_bppt', 'huggingface:id_puisi', 'huggingface:igbo_english_machine_translation', 'huggingface:igbo_monolingual', 'huggingface:igbo_ner', 'huggingface:ilist', 'huggingface:imagenet-1k', 'huggingface:imagenet_sketch', 'huggingface:imdb', 'huggingface:imdb_urdu_reviews', 'huggingface:imppres', 'huggingface:indic_glue', 'huggingface:indonli', 'huggingface:indonlu', 'huggingface:inquisitive_qg', 'huggingface:interpress_news_category_tr', 'huggingface:interpress_news_category_tr_lite', 'huggingface:irc_disentangle', 'huggingface:isixhosa_ner_corpus', 'huggingface:isizulu_ner_corpus', 'huggingface:iwslt2017', 'huggingface:jeopardy', 'huggingface:jfleg', 'huggingface:jigsaw_toxicity_pred', 'huggingface:jigsaw_unintended_bias', 'huggingface:jnlpba', 'huggingface:journalists_questions', 'huggingface:kan_hope', 'huggingface:kannada_news', 'huggingface:kd_conv', 'huggingface:kde4', 'huggingface:kelm', 'huggingface:kilt_tasks', 'huggingface:kilt_wikipedia', 'huggingface:kinnews_kirnews', 'huggingface:klue', 'huggingface:kor_3i4k', 'huggingface:kor_hate', 'huggingface:kor_ner', 'huggingface:kor_nli', 'huggingface:kor_nlu', 'huggingface:kor_qpair', 'huggingface:kor_sae', 'huggingface:kor_sarcasm', 'huggingface:labr', 'huggingface:lama', 'huggingface:lambada', 'huggingface:large_spanish_corpus', 'huggingface:laroseda', 'huggingface:lc_quad', 'huggingface:lccc', 'huggingface:lener_br', 'huggingface:lex_glue', 'huggingface:liar', 'huggingface:librispeech_asr', 'huggingface:librispeech_lm', 'huggingface:limit', 'huggingface:lince', 'huggingface:linnaeus', 'huggingface:liveqa', 'huggingface:lj_speech', 'huggingface:lm1b', 'huggingface:lst20', 'huggingface:m_lama', 'huggingface:mac_morpho', 'huggingface:makhzan', 'huggingface:masakhaner', 'huggingface:math_dataset', 'huggingface:math_qa', 'huggingface:matinf', 'huggingface:mbpp', 'huggingface:mc4', 'huggingface:mc_taco', 'huggingface:md_gender_bias', 'huggingface:mdd', 'huggingface:med_hop', 'huggingface:medal', 'huggingface:medical_dialog', 'huggingface:medical_questions_pairs', 'huggingface:medmcqa', 'huggingface:menyo20k_mt', 'huggingface:meta_woz', 'huggingface:metashift', 'huggingface:metooma', 'huggingface:metrec', 'huggingface:miam', 'huggingface:mkb', 'huggingface:mkqa', 'huggingface:mlqa', 'huggingface:mlsum', 'huggingface:mnist', 'huggingface:mocha', 'huggingface:monash_tsf', 'huggingface:moroco', 'huggingface:movie_rationales', 'huggingface:mrqa', 'huggingface:ms_marco', 'huggingface:ms_terms', 'huggingface:msr_genomics_kbcomp', 'huggingface:msr_sqa', 'huggingface:msr_text_compression', 'huggingface:msr_zhen_translation_parity', 'huggingface:msra_ner', 'huggingface:mt_eng_vietnamese', 'huggingface:muchocine', 'huggingface:multi_booked', 'huggingface:multi_eurlex', 'huggingface:multi_news', 'huggingface:multi_nli', 'huggingface:multi_nli_mismatch', 'huggingface:multi_para_crawl', 'huggingface:multi_re_qa', 'huggingface:multi_woz_v22', 'huggingface:multi_x_science_sum', 'huggingface:multidoc2dial', 'huggingface:multilingual_librispeech', 'huggingface:mutual_friends', 'huggingface:mwsc', 'huggingface:myanmar_news', 'huggingface:narrativeqa', 'huggingface:narrativeqa_manual', 'huggingface:natural_questions', 'huggingface:ncbi_disease', 'huggingface:nchlt', 'huggingface:ncslgr', 'huggingface:nell', 'huggingface:neural_code_search', 'huggingface:news_commentary', 'huggingface:newsgroup', 'huggingface:newsph', 'huggingface:newsph_nli', 'huggingface:newspop', 'huggingface:newsqa', 'huggingface:newsroom', 'huggingface:nkjp-ner', 'huggingface:nli_tr', 'huggingface:nlu_evaluation_data', 'huggingface:norec', 'huggingface:norne', 'huggingface:norwegian_ner', 'huggingface:nq_open', 'huggingface:nsmc', 'huggingface:numer_sense', 'huggingface:numeric_fused_head', 'huggingface:oclar', 'huggingface:offcombr', 'huggingface:offenseval2020_tr', 'huggingface:offenseval_dravidian', 'huggingface:ofis_publik', 'huggingface:ohsumed', 'huggingface:ollie', 'huggingface:omp', 'huggingface:onestop_english', 'huggingface:onestop_qa', 'huggingface:open_subtitles', 'huggingface:openai_humaneval', 'huggingface:openbookqa', 'huggingface:openslr', 'huggingface:openwebtext', 'huggingface:opinosis', 'huggingface:opus100', 'huggingface:opus_books', 'huggingface:opus_dgt', 'huggingface:opus_dogc', 'huggingface:opus_elhuyar', 'huggingface:opus_euconst', 'huggingface:opus_finlex', 'huggingface:opus_fiskmo', 'huggingface:opus_gnome', 'huggingface:opus_infopankki', 'huggingface:opus_memat', 'huggingface:opus_montenegrinsubs', 'huggingface:opus_openoffice', 'huggingface:opus_paracrawl', 'huggingface:opus_rf', 'huggingface:opus_tedtalks', 'huggingface:opus_ubuntu', 'huggingface:opus_wikipedia', 'huggingface:opus_xhosanavy', 'huggingface:orange_sum', 'huggingface:oscar', 'huggingface:para_crawl', 'huggingface:para_pat', 'huggingface:parsinlu_reading_comprehension', 'huggingface:pass', 'huggingface:paws', 'huggingface:paws-x', 'huggingface:pec', 'huggingface:peer_read', 'huggingface:peoples_daily_ner', 'huggingface:per_sent', 'huggingface:persian_ner', 'huggingface:pg19', 'huggingface:php', 'huggingface:piaf', 'huggingface:pib', 'huggingface:piqa', 'huggingface:pn_summary', 'huggingface:poem_sentiment', 'huggingface:polemo2', 'huggingface:poleval2019_cyberbullying', 'huggingface:poleval2019_mt', 'huggingface:polsum', 'huggingface:polyglot_ner', 'huggingface:prachathai67k', 'huggingface:pragmeval', 'huggingface:proto_qa', 'huggingface:psc', 'huggingface:ptb_text_only', 'huggingface:pubmed', 'huggingface:pubmed_qa', 'huggingface:py_ast', 'huggingface:qa4mre', 'huggingface:qa_srl', 'huggingface:qa_zre', 'huggingface:qangaroo', 'huggingface:qanta', 'huggingface:qasc', 'huggingface:qasper', 'huggingface:qed', 'huggingface:qed_amara', 'huggingface:quac', 'huggingface:quail', 'huggingface:quarel', 'huggingface:quartz', 'huggingface:quickdraw', 'huggingface:quora', 'huggingface:quoref', 'huggingface:race', 'huggingface:re_dial', 'huggingface:reasoning_bg', 'huggingface:recipe_nlg', 'huggingface:reclor', 'huggingface:red_caps', 'huggingface:reddit', 'huggingface:reddit_tifu', 'huggingface:refresd', 'huggingface:reuters21578', 'huggingface:riddle_sense', 'huggingface:ro_sent', 'huggingface:ro_sts', 'huggingface:ro_sts_parallel', 'huggingface:roman_urdu', 'huggingface:roman_urdu_hate_speech', 'huggingface:ronec', 'huggingface:ropes', 'huggingface:rotten_tomatoes', 'huggingface:russian_super_glue', 'huggingface:rvl_cdip', 'huggingface:s2orc', 'huggingface:samsum', 'huggingface:sanskrit_classic', 'huggingface:saudinewsnet', 'huggingface:sberquad', 'huggingface:sbu_captions', 'huggingface:scan', 'huggingface:scb_mt_enth_2020', 'huggingface:scene_parse_150', 'huggingface:schema_guided_dstc8', 'huggingface:scicite', 'huggingface:scielo', 'huggingface:scientific_papers', 'huggingface:scifact', 'huggingface:sciq', 'huggingface:scitail', 'huggingface:scitldr', 'huggingface:search_qa', 'huggingface:sede', 'huggingface:selqa', 'huggingface:sem_eval_2010_task_8', 'huggingface:sem_eval_2014_task_1', 'huggingface:sem_eval_2018_task_1', 'huggingface:sem_eval_2020_task_11', 'huggingface:sent_comp', 'huggingface:senti_lex', 'huggingface:senti_ws', 'huggingface:sentiment140', 'huggingface:sepedi_ner', 'huggingface:sesotho_ner_corpus', 'huggingface:setimes', 'huggingface:setswana_ner_corpus', 'huggingface:sharc', 'huggingface:sharc_modified', 'huggingface:sick', 'huggingface:silicone', 'huggingface:simple_questions_v2', 'huggingface:siswati_ner_corpus', 'huggingface:smartdata', 'huggingface:sms_spam', 'huggingface:snips_built_in_intents', 'huggingface:snli', 'huggingface:snow_simplified_japanese_corpus', 'huggingface:so_stacksample', 'huggingface:social_bias_frames', 'huggingface:social_i_qa', 'huggingface:sofc_materials_articles', 'huggingface:sogou_news', 'huggingface:spanish_billion_words', 'huggingface:spc', 'huggingface:species_800', 'huggingface:speech_commands', 'huggingface:spider', 'huggingface:squad', 'huggingface:squad_adversarial', 'huggingface:squad_es', 'huggingface:squad_it', 'huggingface:squad_kor_v1', 'huggingface:squad_kor_v2', 'huggingface:squad_v1_pt', 'huggingface:squad_v2', 'huggingface:squadshifts', 'huggingface:srwac', 'huggingface:sst', 'huggingface:stereoset', 'huggingface:story_cloze', 'huggingface:stsb_mt_sv', 'huggingface:stsb_multi_mt', 'huggingface:style_change_detection', 'huggingface:subjqa', 'huggingface:super_glue', 'huggingface:superb', 'huggingface:svhn', 'huggingface:swag', 'huggingface:swahili', 'huggingface:swahili_news', 'huggingface:swda', 'huggingface:swedish_medical_ner', 'huggingface:swedish_ner_corpus', 'huggingface:swedish_reviews', 'huggingface:swiss_judgment_prediction', 'huggingface:tab_fact', 'huggingface:tamilmixsentiment', 'huggingface:tanzil', 'huggingface:tapaco', 'huggingface:tashkeela', 'huggingface:taskmaster1', 'huggingface:taskmaster2', 'huggingface:taskmaster3', 'huggingface:tatoeba', 'huggingface:ted_hrlr', 'huggingface:ted_iwlst2013', 'huggingface:ted_multi', 'huggingface:ted_talks_iwslt', 'huggingface:telugu_books', 'huggingface:telugu_news', 'huggingface:tep_en_fa_para', 'huggingface:text2log', 'huggingface:textvqa', 'huggingface:thai_toxicity_tweet', 'huggingface:thainer', 'huggingface:thaiqa_squad', 'huggingface:thaisum', 'huggingface:the_pile', 'huggingface:the_pile_books3', 'huggingface:the_pile_openwebtext2', 'huggingface:the_pile_stack_exchange', 'huggingface:tilde_model', 'huggingface:time_dial', 'huggingface:times_of_india_news_headlines', 'huggingface:timit_asr', 'huggingface:tiny_shakespeare', 'huggingface:tlc', 'huggingface:tmu_gfm_dataset', 'huggingface:tne', 'huggingface:told-br', 'huggingface:totto', 'huggingface:trec', 'huggingface:trivia_qa', 'huggingface:truthful_qa', 'huggingface:tsac', 'huggingface:ttc4900', 'huggingface:tunizi', 'huggingface:tuple_ie', 'huggingface:turk', 'huggingface:turkic_xwmt', 'huggingface:turkish_movie_sentiment', 'huggingface:turkish_ner', 'huggingface:turkish_product_reviews', 'huggingface:turkish_shrinked_ner', 'huggingface:turku_ner_corpus', 'huggingface:tweet_eval', 'huggingface:tweet_qa', 'huggingface:tweets_ar_en_parallel', 'huggingface:tweets_hate_speech_detection', 'huggingface:twi_text_c3', 'huggingface:twi_wordsim353', 'huggingface:tydiqa', 'huggingface:ubuntu_dialogs_corpus', 'huggingface:udhr', 'huggingface:um005', 'huggingface:un_ga', 'huggingface:un_multi', 'huggingface:un_pc', 'huggingface:universal_dependencies', 'huggingface:universal_morphologies', 'huggingface:urdu_fake_news', 'huggingface:urdu_sentiment_corpus', 'huggingface:vctk', 'huggingface:visual_genome', 'huggingface:vivos', 'huggingface:web_nlg', 'huggingface:web_of_science', 'huggingface:web_questions', 'huggingface:weibo_ner', 'huggingface:wi_locness', 'huggingface:wider_face', 'huggingface:wiki40b', 'huggingface:wiki_asp', 'huggingface:wiki_atomic_edits', 'huggingface:wiki_auto', 'huggingface:wiki_bio', 'huggingface:wiki_dpr', 'huggingface:wiki_hop', 'huggingface:wiki_lingua', 'huggingface:wiki_movies', 'huggingface:wiki_qa', 'huggingface:wiki_qa_ar', 'huggingface:wiki_snippets', 'huggingface:wiki_source', 'huggingface:wiki_split', 'huggingface:wiki_summary', 'huggingface:wikiann', 'huggingface:wikicorpus', 'huggingface:wikihow', 'huggingface:wikipedia', 'huggingface:wikisql', 'huggingface:wikitablequestions', 'huggingface:wikitext', 'huggingface:wikitext_tl39', 'huggingface:wili_2018', 'huggingface:wino_bias', 'huggingface:winograd_wsc', 'huggingface:winogrande', 'huggingface:wiqa', 'huggingface:wisesight1000', 'huggingface:wisesight_sentiment', 'huggingface:wmt14', 'huggingface:wmt15', 'huggingface:wmt16', 'huggingface:wmt17', 'huggingface:wmt18', 'huggingface:wmt19', 'huggingface:wmt20_mlqe_task1', 'huggingface:wmt20_mlqe_task2', 'huggingface:wmt20_mlqe_task3', 'huggingface:wmt_t2t', 'huggingface:wnut_17', 'huggingface:wongnai_reviews', 'huggingface:woz_dialogue', 'huggingface:wrbsc', 'huggingface:x_stance', 'huggingface:xcopa', 'huggingface:xcsr', 'huggingface:xed_en_fi', 'huggingface:xglue', 'huggingface:xnli', 'huggingface:xor_tydi_qa', 'huggingface:xquad', 'huggingface:xquad_r', 'huggingface:xsum', 'huggingface:xsum_factuality', 'huggingface:xtreme', 'huggingface:yahoo_answers_qa', 'huggingface:yahoo_answers_topics', 'huggingface:yelp_polarity', 'huggingface:yelp_review_full', 'huggingface:yoruba_bbc_topics', 'huggingface:yoruba_gv_ner', 'huggingface:yoruba_text_c3', 'huggingface:yoruba_wordsim353', 'huggingface:youtube_caption_corrections', 'huggingface:zest', 'kubric:kubric_frames', 'kubric:movi_a', 'kubric:movi_b', 'kubric:movi_c', 'kubric:movi_d', 'kubric:movi_e', 'kubric:movi_f', 'kubric:multi_shapenet_frames', 'kubric:nerf_synthetic_frames', 'kubric:nerf_synthetic_scenes', 'kubric:shapenet_pretraining', 'robotics:mt_opt_rlds', 'robotics:mt_opt_sd']\n"
     ]
    }
   ],
   "source": [
    "import tensorflow_datasets as tfds\n",
    "print(tfds.list_builders())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc380d4e",
   "metadata": {
    "id": "cc380d4e"
   },
   "source": [
    "Let's download MNIST. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "cbe12f4f",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "cbe12f4f",
    "outputId": "9d0bc09e-dc42-474b-c261-11dac2b88a34"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(32, 28, 28, 1)\n",
      "[4 1 0 7 8 1 2 7 1 6 6 4 7 7 3 3 7 9 9 1 0 6 6 9 9 4 8 9 4 7 3 3]\n"
     ]
    }
   ],
   "source": [
    "datasets = tfds.load(name=\"mnist\",data_dir='C:/')\n",
    "mnist_train, mnist_test = datasets[\"train\"], datasets[\"test\"]\n",
    "mnist_train = mnist_train.repeat(5).batch(32)\n",
    "mnist_train = mnist_train.map(lambda items: (items[\"image\"], items[\"label\"]))\n",
    "mnist_train = mnist_train.prefetch(1)\n",
    "for images, labels in mnist_train.take(1):\n",
    "    print(images.shape)\n",
    "    print(labels.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17a4dd2f",
   "metadata": {
    "id": "17a4dd2f"
   },
   "source": [
    "Items in the dataset are dictionaries containing both the features and the labels but Keras expects input to be a tuple. We used map() method for transforming the dictionary to a tuple. Additionaly, we can set as_supervised argument to True not to need this dictionary to tuple transformation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "ea67bae8",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ea67bae8",
    "outputId": "8bddcd54-263b-493f-ee12-e10e4f0f82f8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/keras/optimizers/optimizer_v2/gradient_descent.py:108: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  super(SGD, self).__init__(name, **kwargs)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1875/1875 [==============================] - 6s 3ms/step - loss: 32.0200 - accuracy: 0.8425\n",
      "Epoch 2/5\n",
      "1875/1875 [==============================] - 2s 1ms/step - loss: 26.1343 - accuracy: 0.8682\n",
      "Epoch 3/5\n",
      "1875/1875 [==============================] - 2s 1ms/step - loss: 24.9656 - accuracy: 0.8738\n",
      "Epoch 4/5\n",
      "1875/1875 [==============================] - 2s 1ms/step - loss: 24.0894 - accuracy: 0.8759\n",
      "Epoch 5/5\n",
      "1875/1875 [==============================] - 2s 1ms/step - loss: 23.8099 - accuracy: 0.8774\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fe014e860d0>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datasets = tfds.load(name=\"mnist\", batch_size=32, as_supervised=True,data_dir='C:/')\n",
    "mnist_train = datasets[\"train\"].repeat().prefetch(1)\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28, 1]),\n",
    "    keras.layers.Lambda(lambda images: tf.cast(images, tf.float32)),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=[\"accuracy\"])\n",
    "model.fit(mnist_train, steps_per_epoch=60000 // 32, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de1d0cc6",
   "metadata": {
    "id": "de1d0cc6"
   },
   "source": [
    "## Tensorflow Hub"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "127e2985",
   "metadata": {
    "id": "127e2985"
   },
   "source": [
    "Tensorflow Hub is a trained machine learning repository that we can use for implementing trained models.\n",
    "\n",
    "Let's train a text embedding with Tensorflow Hub."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "52bdb1ab",
   "metadata": {
    "id": "52bdb1ab"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "7441f69c",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7441f69c",
    "outputId": "e817bbc9-f780-45fa-a73a-a9c37cbb1a1c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " keras_layer (KerasLayer)    (None, 50)                48190600  \n",
      "                                                                 \n",
      " dense (Dense)               (None, 16)                816       \n",
      "                                                                 \n",
      " dense_1 (Dense)             (None, 1)                 17        \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 48,191,433\n",
      "Trainable params: 833\n",
      "Non-trainable params: 48,190,600\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "import tensorflow_hub as hub\n",
    "\n",
    "hub_layer = hub.KerasLayer(\"https://tfhub.dev/google/tf2-preview/nnlm-en-dim50/1\",\n",
    "                           output_shape=[50], input_shape=[], dtype=tf.string)\n",
    "\n",
    "model = keras.Sequential()\n",
    "model.add(hub_layer)\n",
    "model.add(keras.layers.Dense(16, activation='relu'))\n",
    "model.add(keras.layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "5cdf4b70",
   "metadata": {
    "id": "5cdf4b70"
   },
   "outputs": [],
   "source": [
    "sentences = tf.constant([\"It was a great movie\", \"The actors were amazing\"])\n",
    "embeddings = hub_layer(sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "36428df4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "36428df4",
    "outputId": "0df81a24-85c6-4bf7-8996-8abcb97a1dad"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 50), dtype=float32, numpy=\n",
       "array([[ 7.45939985e-02,  2.76720114e-02,  9.38646123e-02,\n",
       "         1.25124469e-01,  5.40293928e-04, -1.09435350e-01,\n",
       "         1.34755149e-01, -9.57818255e-02, -1.85177118e-01,\n",
       "        -1.69703495e-02,  1.75612606e-02, -9.06603858e-02,\n",
       "         1.12110220e-01,  1.04646273e-01,  3.87700424e-02,\n",
       "        -7.71859884e-02, -3.12189370e-01,  6.99466765e-02,\n",
       "        -4.88970093e-02, -2.99049795e-01,  1.31183028e-01,\n",
       "        -2.12630898e-01,  6.96169436e-02,  1.63592950e-01,\n",
       "         1.05169769e-02,  7.79720694e-02, -2.55230188e-01,\n",
       "        -1.80790052e-01,  2.93739915e-01,  1.62875261e-02,\n",
       "        -2.80566931e-01,  1.60284728e-01,  9.87277832e-03,\n",
       "         8.44555616e-04,  8.39456245e-02,  3.24002892e-01,\n",
       "         1.53253034e-01, -3.01048346e-02,  8.94618109e-02,\n",
       "        -2.39153411e-02, -1.50188789e-01, -1.81733668e-02,\n",
       "        -1.20483577e-01,  1.32937476e-01, -3.35325629e-01,\n",
       "        -1.46504581e-01, -1.25251599e-02, -1.64428815e-01,\n",
       "        -7.00765476e-02,  3.60923223e-02],\n",
       "       [-1.56998575e-01,  4.24599349e-02, -5.57703003e-02,\n",
       "        -8.08446854e-03,  1.23733155e-01,  3.89427543e-02,\n",
       "        -4.37901802e-02, -1.86987907e-01, -2.29341656e-01,\n",
       "        -1.27766818e-01,  3.83025259e-02, -1.07057482e-01,\n",
       "        -6.11584112e-02,  2.49654502e-01, -1.39712945e-01,\n",
       "        -3.91289443e-02, -1.35873526e-01, -3.58613044e-01,\n",
       "         2.53462754e-02, -1.58370987e-01, -1.38350084e-01,\n",
       "        -3.90771806e-01, -6.63642734e-02, -3.24838236e-02,\n",
       "        -2.20453963e-02, -1.68282315e-01, -7.40613639e-02,\n",
       "        -2.49074101e-02,  2.46460736e-01,  9.87201929e-05,\n",
       "        -1.85390845e-01, -4.92824614e-02,  1.09015472e-01,\n",
       "        -9.54203904e-02, -1.60352528e-01, -2.59811729e-02,\n",
       "         1.13778859e-01, -2.09578887e-01,  2.18261331e-01,\n",
       "        -3.11211571e-02, -6.12562597e-02, -8.66057724e-02,\n",
       "        -1.10762455e-01, -5.73977083e-03, -1.08923554e-01,\n",
       "        -1.72919363e-01,  1.00515485e-01, -5.64153939e-02,\n",
       "        -4.97694984e-02, -1.07776590e-01]], dtype=float32)>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embeddings"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}