-1-overview.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Goals\n",
    "\n",
    "- Train a deep neural network!\n",
    "- On not-a-toy-problem\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Bonus Goals\n",
    "- Get `> 80%` accuracy, or\n",
    "- Explain `0.687`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Warnings\n",
    "\n",
    "- Not much theory or explanation\n",
    "- Requires motivation and inquisitiveness\n",
    "- Reading and running someone else's code (aka Programming)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Worse\n",
    "\n",
    "Reading and running a researcher's code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Tools\n",
    "\n",
    "+ `docker`, `conda` & `git` - for reproducible setups\n",
    "    * hopefully, without breaking your computer\n",
    "+ `python3` - programming language of choice\n",
    "+ `jupyter` - reproducible workflows\n",
    "+ `keras` - frontend for deep learning\n",
    "+ `tensorflow` - backend for ML, esp. deep learning\n",
    "+ `numpy` - non-ML numerical processing \n",
    "+ `librosa` - audio processing\n",
    "+ [`audioset`](https://research.google.com/audioset/) - for embeddings (more later)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Programming"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Transforming data from one form to another\n",
    "\n",
    "```\n",
    "INPUTS ---magic---> OUTPUTS\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "where:\n",
    "\n",
    "`magic` is a very specific set of instructions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Audio Segmentation\n",
    "\n",
    "\n",
    "- `INPUT` - audio\n",
    "- `OUTPUT` - frame-wise labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### `magic`\n",
    "\n",
    "- Metric based\n",
    "- Model based (today)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Model Based Audio Segmentation\n",
    "\n",
    "- Acquire and Prepare Training Data\n",
    "    + `TRN`, `VAL`, `TST`\n",
    "    + Keep `TST` + `VAL` to be ~30% of total\n",
    "- Train a suitable model with `TRN`, regularly validating with `VAL`\n",
    "    + Classifier, Regressor, etc.\n",
    "    + Algorithm specific preprocessing might be necessary\n",
    "- Test on `TST` and in your final application scenario"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Acquire and Prepare Data\n",
    "\n",
    "`01-data-acquisition-analysis-splitting.ipynb`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Preprocessing \n",
    "\n",
    "### Feature Extraction\n",
    "\n",
    "Extract [VGGish](https://github.com/tensorflow/models/tree/master/research/audioset) embeddings for each audio in the splits and save separate `tfrecord` for each split.\n",
    "+ Load audio into 16KHz, normalized, single-channel array\n",
    "+ Extract log-mel-spectograms on Hann windows of 0.025s, every 0.010s \n",
    "+ Prepare frames of size 0.96s for VGGish, every 0.12s \n",
    "+ Get 128-dimensional embeddings every 0.12s from VGGish\n",
    "+ Relevant files: \n",
    "    * `02-feature-extraction-vggish_embedding.py`\n",
    "    * `feat_ext.py` - for reusable functions/classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Supervised Training Workflow\n",
    "\n",
    "What we will be doing today\n",
    "\n",
    "[picture](https://goo.gl/images/J4V6mu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- *Make sure your computer won't run out of power*\n",
    "- Duplicate the base notebook `03-training-00-keras_2mlp_clsw0.ipynb`\n",
    "    - Rename to `03-training-<nn>-keras_<model_info>.ipynb`\n",
    "    - Modify and train a [Multi-Layer Perceptron (MLP)](https://www.google.de/search?q=multilayer+perceptron)\n",
    "    - Evaluate on the validation split\n",
    "    - Save learned weights"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}