utf-8''TFDS-Week4-Question.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aKwi1_4l0wev"
   },
   "source": [
    "# Adding a Dataset of Your Own to TFDS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "w9nZyRcLhtiX"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import textwrap\n",
    "import scipy.io\n",
    "import pandas as pd\n",
    "\n",
    "from os import getcwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wooh61rn2FvF"
   },
   "source": [
    "## IMDB Faces Dataset\n",
    "\n",
    "This is the largest publicly available dataset of face images with gender and age labels for training.\n",
    "\n",
    "Source: https://data.vision.ee.ethz.ch/cvl/rrothe/imdb-wiki/\n",
    "\n",
    "The IMDb Faces dataset provides a separate .mat file which can be loaded with Matlab containing all the meta information. The format is as follows:  \n",
    "**dob**: date of birth (Matlab serial date number)  \n",
    "**photo_taken**: year when the photo was taken  \n",
    "**full_path**: path to file  \n",
    "**gender**: 0 for female and 1 for male, NaN if unknown  \n",
    "**name**: name of the celebrity  \n",
    "**face_location**: location of the face (bounding box)  \n",
    "**face_score**: detector score (the higher the better). Inf implies that no face was found in the image and the face_location then just returns the entire image  \n",
    "**second_face_score**: detector score of the face with the second highest score. This is useful to ignore images with more than one face. second_face_score is NaN if no second face was detected.  \n",
    "**celeb_names**: list of all celebrity names  \n",
    "**celeb_id**: index of celebrity name  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's inspect the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uspGC84pWmjR"
   },
   "source": [
    "## Exploring the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "sp7bUzZr3ZUQ"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "imdb.mat\n"
     ]
    }
   ],
   "source": [
    "# Inspect the directory structure\n",
    "imdb_crop_file_path = f\"{getcwd()}/../tmp2/imdb_crop\"\n",
    "files = os.listdir(imdb_crop_file_path)\n",
    "print(textwrap.fill(' '.join(sorted(files)), 80))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1aPlCn9E2PMj"
   },
   "outputs": [],
   "source": [
    "# Inspect the meta data\n",
    "imdb_mat_file_path = f\"{getcwd()}/../tmp2/imdb_crop/imdb.mat\"\n",
    "meta = scipy.io.loadmat(imdb_mat_file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aFj-jsz-6z-I"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'__header__': b'MATLAB 5.0 MAT-file, Platform: GLNXA64, Created on: Sun Jan 17 11:30:27 2016',\n",
       " '__version__': '1.0',\n",
       " '__globals__': [],\n",
       " 'imdb': array([[(array([[693726, 693726, 693726, ..., 726831, 726831, 726831]], dtype=int32), array([[1968, 1970, 1968, ..., 2011, 2011, 2011]], dtype=uint16), array([[array(['01/nm0000001_rm124825600_1899-5-10_1968.jpg'], dtype='<U43'),\n",
       "         array(['01/nm0000001_rm3343756032_1899-5-10_1970.jpg'], dtype='<U44'),\n",
       "         array(['01/nm0000001_rm577153792_1899-5-10_1968.jpg'], dtype='<U43'),\n",
       "         ...,\n",
       "         array(['08/nm3994408_rm926592512_1989-12-29_2011.jpg'], dtype='<U44'),\n",
       "         array(['08/nm3994408_rm943369728_1989-12-29_2011.jpg'], dtype='<U44'),\n",
       "         array(['08/nm3994408_rm976924160_1989-12-29_2011.jpg'], dtype='<U44')]],\n",
       "       dtype=object), array([[1., 1., 1., ..., 0., 0., 0.]]), array([[array(['Fred Astaire'], dtype='<U12'),\n",
       "         array(['Fred Astaire'], dtype='<U12'),\n",
       "         array(['Fred Astaire'], dtype='<U12'), ...,\n",
       "         array(['Jane Levy'], dtype='<U9'),\n",
       "         array(['Jane Levy'], dtype='<U9'),\n",
       "         array(['Jane Levy'], dtype='<U9')]], dtype=object), array([[array([[1072.926,  161.838, 1214.784,  303.696]]),\n",
       "         array([[477.184, 100.352, 622.592, 245.76 ]]),\n",
       "         array([[114.96964309, 114.96964309, 451.68657236, 451.68657236]]),\n",
       "         ..., array([[  1,   1, 453, 640]], dtype=uint16),\n",
       "         array([[144.75225472, 126.76472288, 305.78804127, 287.80050943]]),\n",
       "         array([[457.524,  41.748, 518.016, 102.24 ]])]], dtype=object), array([[1.45969291, 2.5431976 , 3.45557949, ...,       -inf, 4.45072452,\n",
       "         2.13350269]]), array([[1.11897336, 1.85200773, 2.98566022, ...,        nan,        nan,\n",
       "                nan]]), array([[array([\"'Lee' George Quinones\"], dtype='<U21'),\n",
       "         array([\"'Weird Al' Yankovic\"], dtype='<U19'),\n",
       "         array(['2 Chainz'], dtype='<U8'), ...,\n",
       "         array(['Éric Caravaca'], dtype='<U13'),\n",
       "         array(['Ólafur Darri Ólafsson'], dtype='<U21'),\n",
       "         array(['Óscar Jaenada'], dtype='<U13')]], dtype=object), array([[6488, 6488, 6488, ..., 8410, 8410, 8410]], dtype=uint16))]],\n",
       "       dtype=[('dob', 'O'), ('photo_taken', 'O'), ('full_path', 'O'), ('gender', 'O'), ('name', 'O'), ('face_location', 'O'), ('face_score', 'O'), ('second_face_score', 'O'), ('celeb_names', 'O'), ('celeb_id', 'O')])}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rnPmrXJ9XAkK"
   },
   "source": [
    "## Extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zOBtgW6U_VgP"
   },
   "source": [
    "Let's clear up the clutter by going to the metadata's most useful key (imdb) and start exploring all the other keys inside it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "fgrZJWOA7RVa"
   },
   "outputs": [],
   "source": [
    "root = meta['imdb'][0, 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BqqaBw6Y7tku"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('dob', '|O'),\n",
       " ('photo_taken', '|O'),\n",
       " ('full_path', '|O'),\n",
       " ('gender', '|O'),\n",
       " ('name', '|O'),\n",
       " ('face_location', '|O'),\n",
       " ('face_score', '|O'),\n",
       " ('second_face_score', '|O'),\n",
       " ('celeb_names', '|O'),\n",
       " ('celeb_id', '|O')]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "desc = root.dtype.descr\n",
    "desc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "s3WJXw4G2cPk"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Filepaths: [array(['01/nm0000001_rm124825600_1899-5-10_1968.jpg'], dtype='<U43')\n",
      " array(['01/nm0000001_rm3343756032_1899-5-10_1970.jpg'], dtype='<U44')\n",
      " array(['01/nm0000001_rm577153792_1899-5-10_1968.jpg'], dtype='<U43') ...\n",
      " array(['08/nm3994408_rm926592512_1989-12-29_2011.jpg'], dtype='<U44')\n",
      " array(['08/nm3994408_rm943369728_1989-12-29_2011.jpg'], dtype='<U44')\n",
      " array(['08/nm3994408_rm976924160_1989-12-29_2011.jpg'], dtype='<U44')]\n",
      "\n",
      "Names: [array(['Fred Astaire'], dtype='<U12')\n",
      " array(['Fred Astaire'], dtype='<U12')\n",
      " array(['Fred Astaire'], dtype='<U12') ...\n",
      " array(['Jane Levy'], dtype='<U9') array(['Jane Levy'], dtype='<U9')\n",
      " array(['Jane Levy'], dtype='<U9')]\n",
      "\n",
      "Dates of birth: [693726 693726 693726 ... 726831 726831 726831]\n",
      "\n",
      "Genders: [1. 1. 1. ... 0. 0. 0.]\n",
      "\n",
      "Years when the photos were taken: [1968 1970 1968 ... 2011 2011 2011]\n",
      "\n",
      "Face scores: [1.45969291 2.5431976  3.45557949 ...       -inf 4.45072452 2.13350269]\n",
      "\n",
      "Face locations: [array([[1072.926,  161.838, 1214.784,  303.696]])\n",
      " array([[477.184, 100.352, 622.592, 245.76 ]])\n",
      " array([[114.96964309, 114.96964309, 451.68657236, 451.68657236]]) ...\n",
      " array([[  1,   1, 453, 640]], dtype=uint16)\n",
      " array([[144.75225472, 126.76472288, 305.78804127, 287.80050943]])\n",
      " array([[457.524,  41.748, 518.016, 102.24 ]])]\n",
      "\n",
      "Second face scores: [1.11897336 1.85200773 2.98566022 ...        nan        nan        nan]\n",
      "\n",
      "Celeb IDs: [6488 6488 6488 ... 8410 8410 8410]\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# EXERCISE: Fill in the missing code below.\n",
    "\n",
    "full_path = root[\"full_path\"][0]\n",
    "\n",
    "# Do the same for other attributes\n",
    "names = root['name'][0]\n",
    "dob = root['dob'][0]\n",
    "gender = root['gender'][0]\n",
    "photo_taken = root['photo_taken'][0]\n",
    "face_score = root['face_score'][0]\n",
    "face_locations = root['face_location'][0]\n",
    "second_face_score = root['second_face_score'][0]\n",
    "celeb_names = root['celeb_names'][0]\n",
    "celeb_ids = root['celeb_id'][0]\n",
    "\n",
    "print('Filepaths: {}\\n\\n'\n",
    "      'Names: {}\\n\\n'\n",
    "      'Dates of birth: {}\\n\\n'\n",
    "      'Genders: {}\\n\\n'\n",
    "      'Years when the photos were taken: {}\\n\\n'\n",
    "      'Face scores: {}\\n\\n'\n",
    "      'Face locations: {}\\n\\n'\n",
    "      'Second face scores: {}\\n\\n'\n",
    "      'Celeb IDs: {}\\n\\n'\n",
    "      .format(full_path, names, dob, gender, photo_taken, face_score, face_locations, second_face_score, celeb_ids))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zjKXJU1yEnMb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Celeb names: [array([\"'Lee' George Quinones\"], dtype='<U21')\n",
      " array([\"'Weird Al' Yankovic\"], dtype='<U19')\n",
      " array(['2 Chainz'], dtype='<U8') ...\n",
      " array(['Éric Caravaca'], dtype='<U13')\n",
      " array(['Ólafur Darri Ólafsson'], dtype='<U21')\n",
      " array(['Óscar Jaenada'], dtype='<U13')]\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('Celeb names: {}\\n\\n'.format(celeb_names))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TT0un3eFXNW-"
   },
   "source": [
    "Display all the distinct keys and their corresponding values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rYb98AUtC_fA"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['dob',\n",
       " 'photo_taken',\n",
       " 'full_path',\n",
       " 'gender',\n",
       " 'name',\n",
       " 'face_location',\n",
       " 'face_score',\n",
       " 'second_face_score',\n",
       " 'celeb_names',\n",
       " 'celeb_id']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "names = [x[0] for x in desc]\n",
    "names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xJJ9j56hDvnN"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'dob': array([693726, 693726, 693726, ..., 726831, 726831, 726831], dtype=int32),\n",
       " 'photo_taken': array([1968, 1970, 1968, ..., 2011, 2011, 2011], dtype=uint16),\n",
       " 'full_path': array([array(['01/nm0000001_rm124825600_1899-5-10_1968.jpg'], dtype='<U43'),\n",
       "        array(['01/nm0000001_rm3343756032_1899-5-10_1970.jpg'], dtype='<U44'),\n",
       "        array(['01/nm0000001_rm577153792_1899-5-10_1968.jpg'], dtype='<U43'),\n",
       "        ...,\n",
       "        array(['08/nm3994408_rm926592512_1989-12-29_2011.jpg'], dtype='<U44'),\n",
       "        array(['08/nm3994408_rm943369728_1989-12-29_2011.jpg'], dtype='<U44'),\n",
       "        array(['08/nm3994408_rm976924160_1989-12-29_2011.jpg'], dtype='<U44')],\n",
       "       dtype=object),\n",
       " 'gender': array([1., 1., 1., ..., 0., 0., 0.]),\n",
       " 'name': array([array(['Fred Astaire'], dtype='<U12'),\n",
       "        array(['Fred Astaire'], dtype='<U12'),\n",
       "        array(['Fred Astaire'], dtype='<U12'), ...,\n",
       "        array(['Jane Levy'], dtype='<U9'),\n",
       "        array(['Jane Levy'], dtype='<U9'),\n",
       "        array(['Jane Levy'], dtype='<U9')], dtype=object),\n",
       " 'face_location': array([array([[1072.926,  161.838, 1214.784,  303.696]]),\n",
       "        array([[477.184, 100.352, 622.592, 245.76 ]]),\n",
       "        array([[114.96964309, 114.96964309, 451.68657236, 451.68657236]]),\n",
       "        ..., array([[  1,   1, 453, 640]], dtype=uint16),\n",
       "        array([[144.75225472, 126.76472288, 305.78804127, 287.80050943]]),\n",
       "        array([[457.524,  41.748, 518.016, 102.24 ]])], dtype=object),\n",
       " 'face_score': array([1.45969291, 2.5431976 , 3.45557949, ...,       -inf, 4.45072452,\n",
       "        2.13350269]),\n",
       " 'second_face_score': array([1.11897336, 1.85200773, 2.98566022, ...,        nan,        nan,\n",
       "               nan]),\n",
       " 'celeb_names': array([array([\"'Lee' George Quinones\"], dtype='<U21'),\n",
       "        array([\"'Weird Al' Yankovic\"], dtype='<U19'),\n",
       "        array(['2 Chainz'], dtype='<U8'), ...,\n",
       "        array(['Éric Caravaca'], dtype='<U13'),\n",
       "        array(['Ólafur Darri Ólafsson'], dtype='<U21'),\n",
       "        array(['Óscar Jaenada'], dtype='<U13')], dtype=object),\n",
       " 'celeb_id': array([6488, 6488, 6488, ..., 8410, 8410, 8410], dtype=uint16)}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "values = {key: root[key][0] for key in names}\n",
    "values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lYob5mjgXpuy"
   },
   "source": [
    "## Cleanup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3YRjp2gpXbRA"
   },
   "source": [
    "Pop out the celeb names as they are not relevant for creating the records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VRi5bcqnFBua"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'celeb_names'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "del values['celeb_names']\n",
    "names.pop(names.index('celeb_names'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "V2uhpASzXhuy"
   },
   "source": [
    "Let's see how many values are present in each key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4Zu_L_QFEPEm"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dob 460723\n",
      "photo_taken 460723\n",
      "full_path 460723\n",
      "gender 460723\n",
      "name 460723\n",
      "face_location 460723\n",
      "face_score 460723\n",
      "second_face_score 460723\n",
      "celeb_id 460723\n"
     ]
    }
   ],
   "source": [
    "for key, value in values.items():\n",
    "    print(key, len(value))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uJUvw-MBXuKb"
   },
   "source": [
    "## Dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2_uZu2ZQ_169"
   },
   "source": [
    "Now, let's try examining one example from the dataset. To do this, let's load all the attributes that we've extracted just now into a Pandas dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "x-O0pLwWAREq"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dob</th>\n",
       "      <th>photo_taken</th>\n",
       "      <th>full_path</th>\n",
       "      <th>gender</th>\n",
       "      <th>name</th>\n",
       "      <th>face_location</th>\n",
       "      <th>face_score</th>\n",
       "      <th>second_face_score</th>\n",
       "      <th>celeb_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>693726</td>\n",
       "      <td>1968</td>\n",
       "      <td>[01/nm0000001_rm124825600_1899-5-10_1968.jpg]</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[Fred Astaire]</td>\n",
       "      <td>[[1072.926, 161.838, 1214.7839999999999, 303.6...</td>\n",
       "      <td>1.459693</td>\n",
       "      <td>1.118973</td>\n",
       "      <td>6488</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>693726</td>\n",
       "      <td>1970</td>\n",
       "      <td>[01/nm0000001_rm3343756032_1899-5-10_1970.jpg]</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[Fred Astaire]</td>\n",
       "      <td>[[477.184, 100.352, 622.592, 245.76]]</td>\n",
       "      <td>2.543198</td>\n",
       "      <td>1.852008</td>\n",
       "      <td>6488</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>693726</td>\n",
       "      <td>1968</td>\n",
       "      <td>[01/nm0000001_rm577153792_1899-5-10_1968.jpg]</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[Fred Astaire]</td>\n",
       "      <td>[[114.96964308962852, 114.96964308962852, 451....</td>\n",
       "      <td>3.455579</td>\n",
       "      <td>2.985660</td>\n",
       "      <td>6488</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>693726</td>\n",
       "      <td>1968</td>\n",
       "      <td>[01/nm0000001_rm946909184_1899-5-10_1968.jpg]</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[Fred Astaire]</td>\n",
       "      <td>[[622.8855056426588, 424.21750383700805, 844.3...</td>\n",
       "      <td>1.872117</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6488</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>693726</td>\n",
       "      <td>1968</td>\n",
       "      <td>[01/nm0000001_rm980463616_1899-5-10_1968.jpg]</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[Fred Astaire]</td>\n",
       "      <td>[[1013.8590023603723, 233.8820422075853, 1201....</td>\n",
       "      <td>1.158766</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6488</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      dob  photo_taken                                       full_path  \\\n",
       "0  693726         1968   [01/nm0000001_rm124825600_1899-5-10_1968.jpg]   \n",
       "1  693726         1970  [01/nm0000001_rm3343756032_1899-5-10_1970.jpg]   \n",
       "2  693726         1968   [01/nm0000001_rm577153792_1899-5-10_1968.jpg]   \n",
       "3  693726         1968   [01/nm0000001_rm946909184_1899-5-10_1968.jpg]   \n",
       "4  693726         1968   [01/nm0000001_rm980463616_1899-5-10_1968.jpg]   \n",
       "\n",
       "   gender            name                                      face_location  \\\n",
       "0     1.0  [Fred Astaire]  [[1072.926, 161.838, 1214.7839999999999, 303.6...   \n",
       "1     1.0  [Fred Astaire]              [[477.184, 100.352, 622.592, 245.76]]   \n",
       "2     1.0  [Fred Astaire]  [[114.96964308962852, 114.96964308962852, 451....   \n",
       "3     1.0  [Fred Astaire]  [[622.8855056426588, 424.21750383700805, 844.3...   \n",
       "4     1.0  [Fred Astaire]  [[1013.8590023603723, 233.8820422075853, 1201....   \n",
       "\n",
       "   face_score  second_face_score  celeb_id  \n",
       "0    1.459693           1.118973      6488  \n",
       "1    2.543198           1.852008      6488  \n",
       "2    3.455579           2.985660      6488  \n",
       "3    1.872117                NaN      6488  \n",
       "4    1.158766                NaN      6488  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(values, columns=names)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "w-wdFD8uIyRf"
   },
   "source": [
    "The Pandas dataframe may contain some Null values or nan. We will have to filter them later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YGsTHc2VIoJh"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dob                       0\n",
       "photo_taken               0\n",
       "full_path                 0\n",
       "gender                 8462\n",
       "name                      0\n",
       "face_location             0\n",
       "face_score                0\n",
       "second_face_score    246926\n",
       "celeb_id                  0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DS-9rLTR065l"
   },
   "source": [
    "# TensorFlow Datasets\n",
    "\n",
    "TFDS provides a way to transform all those datasets into a standard format, do the preprocessing necessary to make them ready for a machine learning pipeline, and provides a standard input pipeline using `tf.data`.\n",
    "\n",
    "To enable this, each dataset implements a subclass of `DatasetBuilder`, which specifies:\n",
    "\n",
    "* Where the data is coming from (i.e. its URL). \n",
    "* What the dataset looks like (i.e. its features).  \n",
    "* How the data should be split (e.g. TRAIN and TEST). \n",
    "* The individual records in the dataset.\n",
    "\n",
    "The first time a dataset is used, the dataset is downloaded, prepared, and written to disk in a standard format. Subsequent access will read from those pre-processed files directly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6bGCSA-jX0Uw"
   },
   "source": [
    "## Clone the TFDS Repository\n",
    "\n",
    "The next step will be to clone the GitHub TFDS Repository. For this particular notebook, we will clone a particular version of the repository. You can clone the repository by running the following command:\n",
    "\n",
    "```\n",
    "!git clone https://github.com/tensorflow/datasets.git -b v1.2.0\n",
    "```\n",
    "\n",
    "However, for simplicity, we have already cloned this repository for you and placed the files locally. Therefore, there is no need to run the above command if you are running this notebook in Coursera environment.\n",
    "\n",
    "Next, we set the current working directory to `/datasets/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KhYXnLCf5F-Y"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/tf/week4/datasets\n"
     ]
    }
   ],
   "source": [
    "cd datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6Fct97VEYxlT"
   },
   "source": [
    "If you want to contribute to TFDS' repo and add a new dataset, you can use the the following script to help you generate a template of the required python file. To use it, you must first clone the tfds repository and then run the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wZ3psFN65G9u"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset generated in /usr/local/lib/python3.6/dist-packages/tensorflow_datasets\n",
      "You can start with searching TODO(my_dataset).\n",
      "Please check this `https://github.com/tensorflow/datasets/blob/master/docs/add_dataset.md`for details.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "python tensorflow_datasets/scripts/create_new_dataset.py \\\n",
    "  --dataset my_dataset \\\n",
    "  --type image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "a5UbwBVRTmb2"
   },
   "source": [
    "If you wish to see the template generated by the `create_new_dataset.py` file, navigate to the folder indicated in the above cell output. Then go to the `/image/` folder and look for a file called `my_dataset.py`. Feel free to open the file and inspect it. You will see a template with place holders, indicated with the word `TODO`, where you have to fill in the information. \n",
    "\n",
    "Now we will use IPython's `%%writefile` in-built magic command to write whatever is in the current cell into a file. To create or overwrite a file you can use:\n",
    "```\n",
    "%%writefile filename\n",
    "```\n",
    "\n",
    "Let's see an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qkspG9KV7X7i"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting something.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile something.py\n",
    "x = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TQ--c2h0K6R1"
   },
   "source": [
    "Now that the file has been written, let's inspect its contents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VqBEa9UrK4-Z"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x = 10\r\n"
     ]
    }
   ],
   "source": [
    "!cat something.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UJT2Mh-bYmYa"
   },
   "source": [
    "## Define the Dataset with `GeneratorBasedBuilder`\n",
    "\n",
    "Most datasets subclass `tfds.core.GeneratorBasedBuilder`, which is a subclass of `tfds.core.DatasetBuilder` that simplifies defining a dataset. It works well for datasets that can be generated on a single machine. Its subclasses implement:\n",
    "\n",
    "* `_info`: builds the DatasetInfo object describing the dataset\n",
    "\n",
    "\n",
    "* `_split_generators`: downloads the source data and defines the dataset splits\n",
    "\n",
    "\n",
    "* `_generate_examples`: yields (key, example) tuples in the dataset from the source data\n",
    "\n",
    "In this exercise, you will use the `GeneratorBasedBuilder`.\n",
    "\n",
    "### EXERCISE: Fill in the missing code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cYyTvIoO7FqS"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting tensorflow_datasets/image/imdb_faces.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile tensorflow_datasets/image/imdb_faces.py\n",
    "\n",
    "# coding=utf-8\n",
    "# Copyright 2019 The TensorFlow Datasets Authors.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "\n",
    "\"\"\"IMDB Faces dataset.\"\"\"\n",
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "import collections\n",
    "import os\n",
    "import re\n",
    "\n",
    "import tensorflow as tf\n",
    "import tensorflow_datasets.public_api as tfds\n",
    "\n",
    "_DESCRIPTION = \"\"\"Since the publicly available face image datasets are often of small to medium size, rarely exceeding tens of thousands of images, this is an attempt to put together a diverse dataset in that domain.\"\"\"\n",
    "\n",
    "_URL = (\"https://data.vision.ee.ethz.ch/cvl/rrothe/imdb-wiki/\")\n",
    "_DATASET_ROOT_DIR = 'imdb_crop'\n",
    "_ANNOTATION_FILE = 'imdb.mat'\n",
    "\n",
    "\n",
    "_CITATION = \"\"\"@article{Rothe-IJCV-2016,\n",
    "  author = {Rasmus Rothe and Radu Timofte and Luc Van Gool},\n",
    "  title = {Deep expectation of real and apparent age from a single image without facial landmarks},\n",
    "  journal = {International Journal of Computer Vision (IJCV)},\n",
    "  year = {2016},\n",
    "  month = {July},\n",
    "}\n",
    "@InProceedings{Rothe-ICCVW-2015,\n",
    "  author = {Rasmus Rothe and Radu Timofte and Luc Van Gool},\n",
    "  title = {DEX: Deep EXpectation of apparent age from a single image},\n",
    "  booktitle = {IEEE International Conference on Computer Vision Workshops (ICCVW)},\n",
    "  year = {2015},\n",
    "  month = {December},\n",
    "}\n",
    "\"\"\"\n",
    "\n",
    "# Source URL of the IMDB faces dataset\n",
    "_TARBALL_URL = \"https://data.vision.ee.ethz.ch/cvl/rrothe/imdb-wiki/static/imdb_crop.tar\"\n",
    "\n",
    "class ImdbFaces(tfds.core.GeneratorBasedBuilder):\n",
    "  \"\"\"IMDB Faces dataset.\"\"\"\n",
    "\n",
    "  VERSION = tfds.core.Version(\"0.1.0\")\n",
    "  \n",
    "  def _info(self):\n",
    "    return tfds.core.DatasetInfo(\n",
    "        builder=self,\n",
    "        description=_DESCRIPTION,\n",
    "        # Describe the features of the dataset by following this url\n",
    "        # https://www.tensorflow.org/datasets/api_docs/python/tfds/features\n",
    "        features=tfds.features.FeaturesDict({\n",
    "            \"image\": tfds.features.Image(),\n",
    "            \"gender\":  tfds.features.ClassLabel(num_classes=2),\n",
    "            \"dob\": tf.int32,\n",
    "            \"photo_taken\": tf.int32,\n",
    "            \"face_location\": tfds.features.BBoxFeature(),\n",
    "            \"face_score\": tf.float32,\n",
    "            \"second_face_score\": tf.float32,\n",
    "            \"celeb_id\": tf.int32\n",
    "        }),\n",
    "        supervised_keys=(\"image\", \"gender\"),\n",
    "        urls=[_URL],\n",
    "        citation=_CITATION)\n",
    "\n",
    "  def _split_generators(self, dl_manager):\n",
    "    # Download the dataset and then extract it.\n",
    "    download_path = dl_manager.download([_TARBALL_URL])\n",
    "    extracted_path = dl_manager.download_and_extract([_TARBALL_URL])\n",
    "\n",
    "    # Parsing the mat file which contains the list of train images\n",
    "    def parse_mat_file(file_name):\n",
    "      with tf.io.gfile.GFile(file_name, \"rb\") as f:\n",
    "        # Add a lazy import for scipy.io and import the loadmat method to \n",
    "        # load the annotation file\n",
    "        dataset = tfds.core.lazy_imports.scipy.io.loadmat(file_name)['imdb']\n",
    "      return dataset\n",
    "\n",
    "    # Parsing the mat file by using scipy's loadmat method\n",
    "    # Pass the path to the annotation file using the downloaded/extracted paths above\n",
    "    meta = parse_mat_file(os.path.join(extracted_path[0], _DATASET_ROOT_DIR, _ANNOTATION_FILE))\n",
    "\n",
    "    # Get the names of celebrities from the metadata\n",
    "    celeb_names = meta[0, 0]['celeb_names'][0]\n",
    "\n",
    "    # Create tuples out of the distinct set of genders and celeb names\n",
    "    self.info.features['gender'].names = ('Female', 'Male')\n",
    "    self.info.features['celeb_id'].names = tuple([x[0] for x in celeb_names])\n",
    "\n",
    "    return [\n",
    "        tfds.core.SplitGenerator(\n",
    "            name=tfds.Split.TRAIN,\n",
    "            gen_kwargs={\n",
    "                \"image_dir\": extracted_path[0],\n",
    "                \"metadata\": meta,\n",
    "            })\n",
    "    ]\n",
    "\n",
    "  def _get_bounding_box_values(self, bbox_annotations, img_width, img_height):\n",
    "    \"\"\"Function to get normalized bounding box values.\n",
    "\n",
    "    Args:\n",
    "      bbox_annotations: list of bbox values in kitti format\n",
    "      img_width: image width\n",
    "      img_height: image height\n",
    "\n",
    "    Returns:\n",
    "      Normalized bounding box xmin, ymin, xmax, ymax values\n",
    "    \"\"\"\n",
    "\n",
    "    ymin = bbox_annotations[0] / img_height\n",
    "    xmin = bbox_annotations[1] / img_width\n",
    "    ymax = bbox_annotations[2] / img_height\n",
    "    xmax = bbox_annotations[3] / img_width\n",
    "    return ymin, xmin, ymax, xmax\n",
    "  \n",
    "  def _get_image_shape(self, image_path):\n",
    "    image = tf.io.read_file(image_path)\n",
    "    image = tf.image.decode_image(image, channels=3)\n",
    "    shape = image.shape[:2]\n",
    "    return shape\n",
    "\n",
    "  def _generate_examples(self, image_dir, metadata):\n",
    "    # Add a lazy import for pandas here (pd)\n",
    "    pd = tfds.core.lazy_imports.pandas\n",
    "\n",
    "    # Extract the root dictionary from the metadata so that you can query all the keys inside it\n",
    "    root = metadata[0, 0]\n",
    "\n",
    "    \"\"\"Extract image names, dobs, genders,  \n",
    "               face locations, \n",
    "               year when the photos were taken,\n",
    "               face scores (second face score too),\n",
    "               celeb ids\n",
    "    \"\"\"\n",
    "    image_names = root[\"full_path\"][0]\n",
    "    # Do the same for other attributes (dob, genders etc)\n",
    "    dobs = root['dob'][0]\n",
    "    genders = root['gender'][0]\n",
    "    face_locations = root['face_location'][0]\n",
    "    photo_taken_years = root['photo_taken'][0]\n",
    "    face_scores = root['face_score'][0]\n",
    "    second_face_scores = root['second_face_score'][0]\n",
    "    celeb_id = root['celeb_id'][0]\n",
    "        \n",
    "    # Now create a dataframe out of all the features like you've seen before\n",
    "    df = pd.DataFrame(\n",
    "        list(zip(image_names, \n",
    "                dobs,\n",
    "                genders,\n",
    "                face_locations,\n",
    "                photo_taken_years,\n",
    "                face_scores,\n",
    "                second_face_scores,\n",
    "                celeb_id)),\n",
    "        columns=['image_names', 'dobs', 'genders', 'face_locations', 'photo_taken_years',\n",
    "                'face_scores', 'second_face_scores', 'celeb_id'])\n",
    "\n",
    "    # Filter dataframe by only having the rows with face_scores > 1.0\n",
    "    df = df[df['face_scores'] > 1.0]\n",
    "\n",
    "\n",
    "    # Remove any records that contain Nulls/NaNs by checking for NaN with .isna()\n",
    "    df = df[~df['genders'].isna()]\n",
    "    df = df[~df['second_face_scores'].isna()]\n",
    "\n",
    "    # Cast genders to integers so that mapping can take place\n",
    "    df.genders = df.genders.astype(int)\n",
    "\n",
    "    # Iterate over all the rows in the dataframe and map each feature\n",
    "    for _, row in df.iterrows():\n",
    "      # Extract filename, gender, dob, photo_taken, \n",
    "      # face_score, second_face_score and celeb_id\n",
    "      filename = os.path.join(image_dir, _DATASET_ROOT_DIR, row['image_names'][0])\n",
    "      gender = row['genders']\n",
    "      dob = row['dobs']\n",
    "      photo_taken = row['photo_taken_years']\n",
    "      face_score = row['face_scores']\n",
    "      second_face_score = row['second_face_scores']\n",
    "      celeb_id = root['celeb_id']\n",
    "\n",
    "      # Get the image shape\n",
    "      image_width, image_height = self._get_image_shape(filename)\n",
    "      # Normalize the bounding boxes by using the face coordinates and the image shape\n",
    "      bbox = self._get_bounding_box_values(row['face_locations'][0], \n",
    "                                           image_width, image_height)\n",
    "\n",
    "      # Yield a feature dictionary \n",
    "      yield filename, {\n",
    "          \"image\": filename,\n",
    "          \"gender\": gender,\n",
    "          \"dob\": dob,\n",
    "          \"photo_taken\": photo_taken,\n",
    "          \"face_location\": tfds.features.BBox(\n",
    "                          ymin=min(bbox[0], 1),\n",
    "                          xmin=min(bbox[0], 1),\n",
    "                          ymax=min(bbox[0], 1),\n",
    "                          xmax=min(bbox[0], 1)\n",
    "          ),\n",
    "          \"face_score\": face_score,\n",
    "          \"second_face_score\": second_face_score,\n",
    "          \"celeb_id\": celeb_id\n",
    "      }\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7Lu65xXYZC8m"
   },
   "source": [
    "## Add an Import for Registration\n",
    "\n",
    "All subclasses of `tfds.core.DatasetBuilder` are automatically registered when their module is imported such that they can be accessed through `tfds.builder` and `tfds.load`.\n",
    "\n",
    "If you're contributing the dataset to `tensorflow/datasets`, you must add the module import to its subdirectory's `__init__.py` (e.g. `image/__init__.py`), as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pKC49eVJXJLe"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting tensorflow_datasets/image/__init__.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile tensorflow_datasets/image/__init__.py\n",
    "# coding=utf-8\n",
    "# Copyright 2019 The TensorFlow Datasets Authors.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "\n",
    "\"\"\"Image datasets.\"\"\"\n",
    "\n",
    "from tensorflow_datasets.image.abstract_reasoning import AbstractReasoning\n",
    "from tensorflow_datasets.image.aflw2k3d import Aflw2k3d\n",
    "from tensorflow_datasets.image.bigearthnet import Bigearthnet\n",
    "from tensorflow_datasets.image.binarized_mnist import BinarizedMNIST\n",
    "from tensorflow_datasets.image.binary_alpha_digits import BinaryAlphaDigits\n",
    "from tensorflow_datasets.image.caltech import Caltech101\n",
    "from tensorflow_datasets.image.caltech_birds import CaltechBirds2010\n",
    "from tensorflow_datasets.image.cats_vs_dogs import CatsVsDogs\n",
    "from tensorflow_datasets.image.cbis_ddsm import CuratedBreastImagingDDSM\n",
    "from tensorflow_datasets.image.celeba import CelebA\n",
    "from tensorflow_datasets.image.celebahq import CelebAHq\n",
    "from tensorflow_datasets.image.chexpert import Chexpert\n",
    "from tensorflow_datasets.image.cifar import Cifar10\n",
    "from tensorflow_datasets.image.cifar import Cifar100\n",
    "from tensorflow_datasets.image.cifar10_corrupted import Cifar10Corrupted\n",
    "from tensorflow_datasets.image.clevr import CLEVR\n",
    "from tensorflow_datasets.image.coco import Coco\n",
    "from tensorflow_datasets.image.coco2014_legacy import Coco2014\n",
    "from tensorflow_datasets.image.coil100 import Coil100\n",
    "from tensorflow_datasets.image.colorectal_histology import ColorectalHistology\n",
    "from tensorflow_datasets.image.colorectal_histology import ColorectalHistologyLarge\n",
    "from tensorflow_datasets.image.cycle_gan import CycleGAN\n",
    "from tensorflow_datasets.image.deep_weeds import DeepWeeds\n",
    "from tensorflow_datasets.image.diabetic_retinopathy_detection import DiabeticRetinopathyDetection\n",
    "from tensorflow_datasets.image.downsampled_imagenet import DownsampledImagenet\n",
    "from tensorflow_datasets.image.dsprites import Dsprites\n",
    "from tensorflow_datasets.image.dtd import Dtd\n",
    "from tensorflow_datasets.image.eurosat import Eurosat\n",
    "from tensorflow_datasets.image.flowers import TFFlowers\n",
    "from tensorflow_datasets.image.food101 import Food101\n",
    "from tensorflow_datasets.image.horses_or_humans import HorsesOrHumans\n",
    "from tensorflow_datasets.image.image_folder import ImageLabelFolder\n",
    "from tensorflow_datasets.image.imagenet import Imagenet2012\n",
    "from tensorflow_datasets.image.imagenet2012_corrupted import Imagenet2012Corrupted\n",
    "from tensorflow_datasets.image.kitti import Kitti\n",
    "from tensorflow_datasets.image.lfw import LFW\n",
    "from tensorflow_datasets.image.lsun import Lsun\n",
    "from tensorflow_datasets.image.mnist import EMNIST\n",
    "from tensorflow_datasets.image.mnist import FashionMNIST\n",
    "from tensorflow_datasets.image.mnist import KMNIST\n",
    "from tensorflow_datasets.image.mnist import MNIST\n",
    "from tensorflow_datasets.image.mnist_corrupted import MNISTCorrupted\n",
    "from tensorflow_datasets.image.omniglot import Omniglot\n",
    "from tensorflow_datasets.image.open_images import OpenImagesV4\n",
    "from tensorflow_datasets.image.oxford_flowers102 import OxfordFlowers102\n",
    "from tensorflow_datasets.image.oxford_iiit_pet import OxfordIIITPet\n",
    "from tensorflow_datasets.image.patch_camelyon import PatchCamelyon\n",
    "from tensorflow_datasets.image.pet_finder import PetFinder\n",
    "from tensorflow_datasets.image.quickdraw import QuickdrawBitmap\n",
    "from tensorflow_datasets.image.resisc45 import Resisc45\n",
    "from tensorflow_datasets.image.rock_paper_scissors import RockPaperScissors\n",
    "from tensorflow_datasets.image.scene_parse_150 import SceneParse150\n",
    "from tensorflow_datasets.image.shapes3d import Shapes3d\n",
    "from tensorflow_datasets.image.smallnorb import Smallnorb\n",
    "from tensorflow_datasets.image.so2sat import So2sat\n",
    "from tensorflow_datasets.image.stanford_dogs import StanfordDogs\n",
    "from tensorflow_datasets.image.stanford_online_products import StanfordOnlineProducts\n",
    "from tensorflow_datasets.image.sun import Sun397\n",
    "from tensorflow_datasets.image.svhn import SvhnCropped\n",
    "from tensorflow_datasets.image.uc_merced import UcMerced\n",
    "from tensorflow_datasets.image.visual_domain_decathlon import VisualDomainDecathlon\n",
    "\n",
    "# EXERCISE: Import your dataset module here\n",
    "\n",
    "from tensorflow_datasets.image.imdb_faces import ImdbFaces"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QYmgS2SrYXtP"
   },
   "source": [
    "## URL Checksums\n",
    "\n",
    "If you're contributing the dataset to `tensorflow/datasets`, add a checksums file for the dataset. On first download, the DownloadManager will automatically add the sizes and checksums for all downloaded URLs to that file. This ensures that on subsequent data generation, the downloaded files are as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cvrp-iHuYG_e"
   },
   "outputs": [],
   "source": [
    "!touch tensorflow_datasets/url_checksums/imdb_faces.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JwnUAn49U-U8"
   },
   "source": [
    "## Build the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Y8uKiqWrU_C0"
   },
   "outputs": [],
   "source": [
    "# EXERCISE: Fill in the name of your dataset.\n",
    "# The name must be a string.\n",
    "DATASET_NAME = \"imdb_faces\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "S7evoTtpon7I"
   },
   "source": [
    "We then run the `download_and_prepare` script locally to build it, using the following command:\n",
    "\n",
    "```\n",
    "%%bash -s $DATASET_NAME\n",
    "python -m tensorflow_datasets.scripts.download_and_prepare \\\n",
    "  --register_checksums \\\n",
    "  --datasets=$1\n",
    "```\n",
    "\n",
    "**NOTE:** It may take more than 30 minutes to download the dataset and then write all the preprocessed files as TFRecords. Due to the enormous size of the data involved, we are unable to run the above script in the Coursera environment. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7hNPD2rraN5o"
   },
   "source": [
    "## Load the Dataset\n",
    "\n",
    "Once the dataset is built you can load it in the usual way, by using `tfds.load`, as shown below:\n",
    "\n",
    "```python\n",
    "import tensorflow_datasets as tfds\n",
    "dataset, info = tfds.load('imdb_faces', with_info=True)\n",
    "```\n",
    "\n",
    "**Note:** Since we couldn't build the `imdb_faces` dataset due to its size, we are unable to run the above code in the Coursera environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore the Dataset\n",
    "\n",
    "Once the dataset is loaded, you can explore it by using the following loop:\n",
    "\n",
    "```python\n",
    "for feature in tfds.as_numpy(dataset['train']):\n",
    "  for key, value in feature.items():\n",
    "    if key == 'image':\n",
    "      value = value.shape\n",
    "    print(key, value)\n",
    "  break\n",
    "```\n",
    "\n",
    "**Note:** Since we couldn't build the `imdb_faces` dataset due to its size, we are unable to run the above code in the Coursera environment.\n",
    "\n",
    "The expected output from the code block shown above should be:\n",
    "\n",
    "```python\n",
    ">>>\n",
    "celeb_id 12387\n",
    "dob 722957\n",
    "face_location [1.         0.56327355 1.         1.        ]\n",
    "face_score 4.0612864\n",
    "gender 0\n",
    "image (96, 97, 3)\n",
    "photo_taken 2007\n",
    "second_face_score 3.6680346\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BhUO2vXDZw8q"
   },
   "source": [
    "# Next steps for publishing\n",
    "\n",
    "**Double-check the citation**  \n",
    "\n",
    "It's important that DatasetInfo.citation includes a good citation for the dataset. It's hard and important work contributing a dataset to the community and we want to make it easy for dataset users to cite the work.\n",
    "\n",
    "If the dataset's website has a specifically requested citation, use that (in BibTex format).\n",
    "\n",
    "If the paper is on arXiv, find it there and click the bibtex link on the right-hand side.\n",
    "\n",
    "If the paper is not on arXiv, find the paper on Google Scholar and click the double-quotation mark underneath the title and on the popup, click BibTeX.\n",
    "\n",
    "If there is no associated paper (for example, there's just a website), you can use the BibTeX Online Editor to create a custom BibTeX entry (the drop-down menu has an Online entry type).\n",
    "  \n",
    "\n",
    "**Add a test**   \n",
    "\n",
    "Most datasets in TFDS should have a unit test and your reviewer may ask you to add one if you haven't already. See the testing section below.   \n",
    "**Check your code style**  \n",
    "\n",
    "Follow the PEP 8 Python style guide, except TensorFlow uses 2 spaces instead of 4. Please conform to the Google Python Style Guide,\n",
    "\n",
    "Most importantly, use tensorflow_datasets/oss_scripts/lint.sh to ensure your code is properly formatted. For example, to lint the image directory\n",
    "See TensorFlow code style guide for more information.\n",
    "\n",
    "**Add release notes**\n",
    "Add the dataset to the release notes. The release note will be published for the next release.\n",
    "\n",
    "**Send for review!**\n",
    "Send the pull request for review.\n",
    "\n",
    "For more information, visit https://www.tensorflow.org/datasets/add_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Submission Instructions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now click the 'Submit Assignment' button above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# When you're done or would like to take a break, please run the two cells below to save your work and close the Notebook. This frees up resources for your fellow learners."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%javascript\n",
    "<!-- Save the notebook -->\n",
    "IPython.notebook.save_checkpoint();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%javascript\n",
    "<!-- Shutdown and close the notebook -->\n",
    "window.onbeforeunload = null\n",
    "window.close();\n",
    "IPython.notebook.session.delete();"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "TFDS Week 4 - Question.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "coursera": {
   "course_slug": "data-pipelines-tensorflow",
   "graded_item_id": "fqOvf",
   "launcher_item_id": "QCJEw"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}