examples/Audio_Classification_with_TensorFlow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9f910b0b",
   "metadata": {
    "id": "9f910b0b"
   },
   "source": [
    "# Audio Classification with TensorFlow\n",
    "\n",
    "Train a CNN based classifier with __TensorFlow__ on Spoken Digit dataset\n",
    "\n",
    "### Typical Audio Classification Approach\n",
    "\n",
    "Typical approach for audio classification would look like this:\n",
    "\n",
    "1. Gather audio data\n",
    "2. Convert audio to frequency domain representation like MFCC or Mel Spectrogram\n",
    "3. Train a CNN on the frequency domain feature\n",
    "4. Deploy the model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hgbTULPihLED",
   "metadata": {
    "id": "hgbTULPihLED"
   },
   "source": [
    "### Setup\n",
    "\n",
    "Install required packages: `teal` and `pydub`\n",
    "\n",
    "`pydub` is needed for the dataset downloaded from TFDS and `teal` is a library of audio specific layers for TensorFlow\n",
    "\n",
    "If you're using Colab, these are the only two packages that are needed as TensorFlow and TFDS are already installed on Google Colab, otherwise these 2 must be installed as well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6278c32",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 14447,
     "status": "ok",
     "timestamp": 1644224327371,
     "user": {
      "displayName": "Amit Yadav",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhPlsR49pVenE006oG1QDmA4uQZFm5zND0AfTfoPA=s64",
      "userId": "06019965579715007028"
     },
     "user_tz": 0
    },
    "id": "f6278c32",
    "outputId": "0885155a-e25f-42be-e416-9f8260999384"
   },
   "outputs": [],
   "source": [
    "!pip install git+https://github.com/am1tyadav/teal pydub -q"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0057dcef",
   "metadata": {
    "id": "0057dcef"
   },
   "source": [
    "Restart kernel for installation to take effect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05d82cf5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 276,
     "status": "ok",
     "timestamp": 1644224330602,
     "user": {
      "displayName": "Amit Yadav",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhPlsR49pVenE006oG1QDmA4uQZFm5zND0AfTfoPA=s64",
      "userId": "06019965579715007028"
     },
     "user_tz": 0
    },
    "id": "05d82cf5",
    "outputId": "c1df8d13-3243-450e-ef01-b1830a0bb675"
   },
   "outputs": [],
   "source": [
    "import IPython\n",
    "\n",
    "IPython.Application.instance().kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "754a2204",
   "metadata": {
    "id": "754a2204"
   },
   "source": [
    "Import TensorFlow and Teal after the kernel restarts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efbee401",
   "metadata": {
    "executionInfo": {
     "elapsed": 4807,
     "status": "ok",
     "timestamp": 1644224342090,
     "user": {
      "displayName": "Amit Yadav",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhPlsR49pVenE006oG1QDmA4uQZFm5zND0AfTfoPA=s64",
      "userId": "06019965579715007028"
     },
     "user_tz": 0
    },
    "id": "efbee401"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras import layers, models\n",
    "from IPython.display import Audio\n",
    "import tensorflow_datasets as tfds\n",
    "import teal"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f2e487c",
   "metadata": {
    "id": "2f2e487c"
   },
   "source": [
    "### Download the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "Dr27tSnwWxiu",
   "metadata": {
    "executionInfo": {
     "elapsed": 210,
     "status": "ok",
     "timestamp": 1644224839878,
     "user": {
      "displayName": "Amit Yadav",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhPlsR49pVenE006oG1QDmA4uQZFm5zND0AfTfoPA=s64",
      "userId": "06019965579715007028"
     },
     "user_tz": 0
    },
    "id": "Dr27tSnwWxiu"
   },
   "outputs": [],
   "source": [
    "dataset = tfds.load(\"spoken_digit\", data_dir=\"./tmp\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "288e2ed1",
   "metadata": {},
   "source": [
    "Very few samples in this dataset are longer than 1 second, so we can crop accordingly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b99f752",
   "metadata": {
    "executionInfo": {
     "elapsed": 2,
     "status": "ok",
     "timestamp": 1644224839879,
     "user": {
      "displayName": "Amit Yadav",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhPlsR49pVenE006oG1QDmA4uQZFm5zND0AfTfoPA=s64",
      "userId": "06019965579715007028"
     },
     "user_tz": 0
    },
    "id": "0b99f752"
   },
   "outputs": [],
   "source": [
    "SAMPLE_RATE = 8000\n",
    "DURATION = 1\n",
    "SAMPLE_LEN = DURATION * SAMPLE_RATE\n",
    "\n",
    "\n",
    "def process_example(example):\n",
    "    audio = example[\"audio\"]\n",
    "    audio = tf.cast(audio, dtype=tf.float32) / 32768.\n",
    "    label = tf.cast(example[\"label\"], dtype=tf.float32)\n",
    "\n",
    "    num_samples = tf.shape(audio)[0]\n",
    "    if num_samples > SAMPLE_LEN:\n",
    "        return audio[:SAMPLE_LEN], label\n",
    "    \n",
    "    # Otherwise pad audio \n",
    "    padding = SAMPLE_LEN - num_samples\n",
    "    \n",
    "    if padding == 0:\n",
    "        return audio, label\n",
    "\n",
    "    return tf.pad(audio, [[0, padding]]), label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb91cd38",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 2,
     "status": "ok",
     "timestamp": 1644224840356,
     "user": {
      "displayName": "Amit Yadav",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhPlsR49pVenE006oG1QDmA4uQZFm5zND0AfTfoPA=s64",
      "userId": "06019965579715007028"
     },
     "user_tz": 0
    },
    "id": "bb91cd38",
    "outputId": "4865042d-0e34-40d5-8278-dea854c44b82"
   },
   "outputs": [],
   "source": [
    "BATCH_SIZE = 8\n",
    "TOTAL_EXAMPLES = len(dataset[\"train\"])\n",
    "NUM_TRAIN = int(0.7 * TOTAL_EXAMPLES)\n",
    "NUM_VALID = TOTAL_EXAMPLES - NUM_TRAIN\n",
    "\n",
    "dataset = dataset[\"train\"]\n",
    "dataset = dataset.map(process_example)\n",
    "\n",
    "print(f\"Splitting dataset into {NUM_TRAIN} training examples and {NUM_VALID} validation examples\")\n",
    "\n",
    "dataset = dataset.shuffle(buffer_size=TOTAL_EXAMPLES)\n",
    "train_ds = dataset.take(NUM_TRAIN).batch(BATCH_SIZE)\n",
    "valid_ds = dataset.skip(NUM_TRAIN).batch(BATCH_SIZE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0u94cOaIYsK-",
   "metadata": {
    "executionInfo": {
     "elapsed": 844,
     "status": "ok",
     "timestamp": 1644224841199,
     "user": {
      "displayName": "Amit Yadav",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhPlsR49pVenE006oG1QDmA4uQZFm5zND0AfTfoPA=s64",
      "userId": "06019965579715007028"
     },
     "user_tz": 0
    },
    "id": "0u94cOaIYsK-"
   },
   "outputs": [],
   "source": [
    "audio, label = next(iter(train_ds))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6FC5rjG4Y29r",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 69
    },
    "executionInfo": {
     "elapsed": 7,
     "status": "ok",
     "timestamp": 1644224841199,
     "user": {
      "displayName": "Amit Yadav",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhPlsR49pVenE006oG1QDmA4uQZFm5zND0AfTfoPA=s64",
      "userId": "06019965579715007028"
     },
     "user_tz": 0
    },
    "id": "6FC5rjG4Y29r",
    "outputId": "0963acfa-f7e6-4f92-808a-b3d3d77b525a"
   },
   "outputs": [],
   "source": [
    "print(label)\n",
    "\n",
    "Audio(audio[0], rate=SAMPLE_RATE)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcf18d13",
   "metadata": {
    "id": "fcf18d13"
   },
   "source": [
    "## Models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6032dcac",
   "metadata": {
    "id": "6032dcac"
   },
   "source": [
    "### Feature Model - Log Mel Spectrogram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cc2b68d",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 5,
     "status": "ok",
     "timestamp": 1644224841199,
     "user": {
      "displayName": "Amit Yadav",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhPlsR49pVenE006oG1QDmA4uQZFm5zND0AfTfoPA=s64",
      "userId": "06019965579715007028"
     },
     "user_tz": 0
    },
    "id": "8cc2b68d",
    "outputId": "81a4b94d-9435-4d47-c5d1-ccf9a9d13aa4"
   },
   "outputs": [],
   "source": [
    "N_FFT = 1024\n",
    "HOP_LEN = 256\n",
    "N_MELS = 28\n",
    "\n",
    "\n",
    "feature_model = models.Sequential([\n",
    "    layers.Input(shape=(SAMPLE_LEN, )),\n",
    "    teal.NormalizeAudio(),\n",
    "    teal.AudioToMelSpectrogram(SAMPLE_RATE, N_FFT, HOP_LEN, N_MELS),\n",
    "    teal.PowerToDb(),\n",
    "    teal.NormalizeSpectrum()\n",
    "], name=\"feature_model\")\n",
    "\n",
    "feature_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6b569a1",
   "metadata": {
    "id": "e6b569a1"
   },
   "source": [
    "### Augmentation Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0d0896f",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 206,
     "status": "ok",
     "timestamp": 1644224841682,
     "user": {
      "displayName": "Amit Yadav",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhPlsR49pVenE006oG1QDmA4uQZFm5zND0AfTfoPA=s64",
      "userId": "06019965579715007028"
     },
     "user_tz": 0
    },
    "id": "d0d0896f",
    "outputId": "2821930c-bcdb-4241-c59e-7aed656ec7e3"
   },
   "outputs": [],
   "source": [
    "aug_model = models.Sequential([\n",
    "    layers.Input(shape=(SAMPLE_LEN, )),\n",
    "    teal.InversePolarity(0.5),\n",
    "    teal.RandomGain(0.2),\n",
    "    teal.RandomNoise(0.4),\n",
    "    teal.PitchShift(0.5, shift=50),\n",
    "    teal.RandomGain(0.2)\n",
    "], name=\"augmentation_model\")\n",
    "\n",
    "aug_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c54956bb",
   "metadata": {
    "id": "c54956bb"
   },
   "source": [
    "### CNN Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad35be0f",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 7,
     "status": "ok",
     "timestamp": 1644224841939,
     "user": {
      "displayName": "Amit Yadav",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhPlsR49pVenE006oG1QDmA4uQZFm5zND0AfTfoPA=s64",
      "userId": "06019965579715007028"
     },
     "user_tz": 0
    },
    "id": "ad35be0f",
    "outputId": "9454f014-24fe-4621-97f3-20a69a3f4d27"
   },
   "outputs": [],
   "source": [
    "cnn = models.Sequential([\n",
    "    layers.Input(shape=(28, 28)),\n",
    "    layers.Lambda(lambda x: tf.expand_dims(x, axis=-1)),\n",
    "    layers.Conv2D(32, 3, padding=\"same\", strides=2, activation=\"relu\"),\n",
    "    layers.BatchNormalization(),\n",
    "    layers.Conv2D(64, 3, padding=\"same\", strides=2, activation=\"relu\"),\n",
    "    layers.BatchNormalization(),\n",
    "    layers.Flatten(),\n",
    "    layers.Dense(32, activation=\"relu\"),\n",
    "    layers.Dropout(0.2),\n",
    "    layers.Dense(10, activation=\"softmax\")\n",
    "], name=\"cnn\")\n",
    "\n",
    "cnn.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40b3c939",
   "metadata": {
    "id": "40b3c939"
   },
   "source": [
    "### Composite Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41d34ce4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 455,
     "status": "ok",
     "timestamp": 1644224843511,
     "user": {
      "displayName": "Amit Yadav",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhPlsR49pVenE006oG1QDmA4uQZFm5zND0AfTfoPA=s64",
      "userId": "06019965579715007028"
     },
     "user_tz": 0
    },
    "id": "41d34ce4",
    "outputId": "d6049015-4d3d-4513-e6f9-6639722a4654"
   },
   "outputs": [],
   "source": [
    "_input = layers.Input(shape=(SAMPLE_LEN, ))\n",
    "_data = aug_model(_input)\n",
    "_feature = feature_model(_data)\n",
    "_output = cnn(_feature)\n",
    "\n",
    "model = models.Model(_input, _output, name=\"composite_model\")\n",
    "model.compile(\n",
    "    loss=\"sparse_categorical_crossentropy\",\n",
    "    optimizer=\"adam\",\n",
    "    metrics=[\"accuracy\"],\n",
    "    run_eagerly=True\n",
    ")\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d55ba23",
   "metadata": {
    "id": "1d55ba23"
   },
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "847f6dc8",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 159987,
     "status": "ok",
     "timestamp": 1644225003496,
     "user": {
      "displayName": "Amit Yadav",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhPlsR49pVenE006oG1QDmA4uQZFm5zND0AfTfoPA=s64",
      "userId": "06019965579715007028"
     },
     "user_tz": 0
    },
    "id": "847f6dc8",
    "outputId": "e8c31ded-a63b-4bdb-e406-3e513a3a245f"
   },
   "outputs": [],
   "source": [
    "EPOCHS = 5\n",
    "\n",
    "_ = model.fit(\n",
    "    train_ds,\n",
    "    validation_data=valid_ds,\n",
    "    epochs=EPOCHS,\n",
    "    callbacks=[tf.keras.callbacks.ReduceLROnPlateau(patience=4)]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "794806bb",
   "metadata": {
    "id": "794806bb"
   },
   "source": [
    "## Saving Model for Production\n",
    "\n",
    "You probably don't want the augmentation model/ layers in your deployed model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f4af411",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 11,
     "status": "ok",
     "timestamp": 1644225003497,
     "user": {
      "displayName": "Amit Yadav",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhPlsR49pVenE006oG1QDmA4uQZFm5zND0AfTfoPA=s64",
      "userId": "06019965579715007028"
     },
     "user_tz": 0
    },
    "id": "4f4af411",
    "outputId": "906cc874-ef7a-437d-c2a9-56b877bbea87"
   },
   "outputs": [],
   "source": [
    "_input = model.input\n",
    "\n",
    "_feature = model.layers[2](_input)\n",
    "_output = model.layers[3](_feature)\n",
    "\n",
    "model = models.Model(_input, _output, name=\"prod_model\")\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "815e8b8d",
   "metadata": {
    "id": "815e8b8d"
   },
   "source": [
    "## Get Some Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51a7f274",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 1791,
     "status": "ok",
     "timestamp": 1644225741856,
     "user": {
      "displayName": "Amit Yadav",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhPlsR49pVenE006oG1QDmA4uQZFm5zND0AfTfoPA=s64",
      "userId": "06019965579715007028"
     },
     "user_tz": 0
    },
    "id": "51a7f274",
    "outputId": "2669809f-cd08-4fd5-e845-854015a26830"
   },
   "outputs": [],
   "source": [
    "examples, labels = next(iter(valid_ds))\n",
    "\n",
    "preds = model.predict(examples)\n",
    "\n",
    "tf.argmax(preds, axis=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b82a7ca7",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 6,
     "status": "ok",
     "timestamp": 1644225741857,
     "user": {
      "displayName": "Amit Yadav",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhPlsR49pVenE006oG1QDmA4uQZFm5zND0AfTfoPA=s64",
      "userId": "06019965579715007028"
     },
     "user_tz": 0
    },
    "id": "b82a7ca7",
    "outputId": "15c36e6c-3364-4223-8e3d-2f3fef2057e9"
   },
   "outputs": [],
   "source": [
    "labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a49afd6d",
   "metadata": {
    "id": "a49afd6d"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Audio_Classification_with_TensorFlow.ipynb",
   "provenance": [
    {
     "file_id": "https://github.com/am1tyadav/teal/blob/main/examples/Audio_Classification_with_TensorFlow_and_Teal.ipynb",
     "timestamp": 1644058388107
    }
   ]
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}