demo_colab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o-BliYu7qgAZ"
   },
   "source": [
    "### Install and import required packages\n",
    "\n",
    "**Resemblyzer** - to derive a \"high-level representation\" of a voice through a deep learning model (https://github.com/resemble-ai/Resemblyzer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "H2fLJFL9QQw7",
    "outputId": "c5c54a78-8edd-4d8a-f0c5-cb3718a37292"
   },
   "outputs": [],
   "source": [
    "!pip install resemblyzer\n",
    "#You may be prompted to restart the runtime after the installation. Look out for that in the output below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JfXk_mYTQdRo"
   },
   "outputs": [],
   "source": [
    "from resemblyzer import VoiceEncoder\n",
    "import numpy as np\n",
    "from umap import UMAP\n",
    "import matplotlib.pyplot as plt\n",
    "import librosa\n",
    "import pickle as pkl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AEmIMldRdW1u"
   },
   "source": [
    "### To enable microphone input\n",
    "(source: https://gist.github.com/Anwarvic/e843ed55c550784aa6f78faa505ca3ff)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "B2RvyCwwXWEA"
   },
   "outputs": [],
   "source": [
    "# all imports\n",
    "from io import BytesIO\n",
    "from base64 import b64decode\n",
    "from google.colab import output\n",
    "from IPython.display import Javascript\n",
    "\n",
    "RECORD = \"\"\"\n",
    "const sleep  = time => new Promise(resolve => setTimeout(resolve, time))\n",
    "const b2text = blob => new Promise(resolve => {\n",
    "  const reader = new FileReader()\n",
    "  reader.onloadend = e => resolve(e.srcElement.result)\n",
    "  reader.readAsDataURL(blob)\n",
    "})\n",
    "var record = time => new Promise(async resolve => {\n",
    "  stream = await navigator.mediaDevices.getUserMedia({ audio: true })\n",
    "  recorder = new MediaRecorder(stream)\n",
    "  chunks = []\n",
    "  recorder.ondataavailable = e => chunks.push(e.data)\n",
    "  recorder.start()\n",
    "  await sleep(time)\n",
    "  recorder.onstop = async ()=>{\n",
    "    blob = new Blob(chunks)\n",
    "    text = await b2text(blob)\n",
    "    resolve(text)\n",
    "  }\n",
    "  recorder.stop()\n",
    "})\n",
    "\"\"\"\n",
    "\n",
    "def record(sec=3):\n",
    "  print(\"Speak Now...\")\n",
    "  display(Javascript(RECORD))\n",
    "  sec += 1\n",
    "  s = output.eval_js('record(%d)' % (sec*1000))\n",
    "  print(\"Done Recording !\")\n",
    "  b = b64decode(s.split(',')[1])\n",
    "  with open('audio.wav','wb') as f:\n",
    "    f.write(b)\n",
    "  audio, sr = librosa.load('audio.wav')\n",
    "  return audio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-SKXnyogAtVa"
   },
   "source": [
    "### Clone the github repository to download some required pretrained files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "id": "70FkgwzyVlC2",
    "outputId": "0516fe67-e47c-4437-add9-433e9b66572d"
   },
   "outputs": [],
   "source": [
    "!git clone https://github.com/rohitma38/sing-alike-interactive.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 88
    },
    "id": "QCGYpSxKQn9N",
    "outputId": "82afa272-dfe1-47e4-a02f-1b09b15b46c6"
   },
   "outputs": [],
   "source": [
    "encoder = VoiceEncoder()\n",
    "reducer = pkl.load(open('sing-alike-interactive/reducer.sav','rb'))\n",
    "knn = pkl.load(open('sing-alike-interactive/knn_model.sav','rb'))\n",
    "projs = np.load('sing-alike-interactive/projections.npy')\n",
    "labels = np.load('sing-alike-interactive/labels.npy')\n",
    "\n",
    "singers = ['Arijit', 'Atif', 'Lata', 'Chinmayi']\n",
    "cluster_coords = [(12.3,1.3),(12.3,13),(-4,-2),(-4,16)]\n",
    "\n",
    "def plot_projection(projs_gt, labels_gt, projs_test):\n",
    "  fig = plt.figure()\n",
    "  ax = fig.add_subplot(111)\n",
    "  ax.scatter(projs_gt[:,0],projs_gt[:,1],c=labels)\n",
    "\n",
    "  for i in range(len(singers)):\n",
    "    plt.annotate(singers[i],cluster_coords[i])\n",
    "    \n",
    "  for item in projs_test:\n",
    "    ax.scatter(item[0],item[1],color='black',marker='+')\n",
    "  plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Oupl2aB_iXDo"
   },
   "outputs": [],
   "source": [
    "prediction_history=[]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aaand, time to play!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K3jNVBHbA2NN"
   },
   "source": [
    "Running below cell will record for 10 seconds from your mic and then compare your voice to 4 popular singers that are currently stored in the system - Arijit Singh (red), Atif Aslam (blue), Lata Mangeshkar (green) and Chinmayi (cyan). Your voice sample will appear as an 'x' mark on a 2-d plane containing the 4 clusters of the artists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 336
    },
    "id": "YjOIHb5-Rxhf",
    "outputId": "cda8c3c4-dafd-4ba5-a388-9eb7012b91d8"
   },
   "outputs": [],
   "source": [
    "print('Recording')\n",
    "data = record(10) \n",
    "decoded = data.copy()\n",
    "\n",
    "embed = encoder.embed_utterance(decoded/np.max(decoded))\n",
    "proj = reducer.transform([embed])\n",
    "print('You sound most like: %s!'%singers[knn.predict(proj)[0]])\n",
    "prediction_history.append([proj[0][0],proj[0][1]])\n",
    "plot_projection(projs,labels,prediction_history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run above cell repeatedly to continue playing with this demo!\n",
    "One fun thing to try would be to check how your voice sample's embedding changes when you sound male versus when you sound female.\n",
    "Have fun! :D"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "demo.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}