proof_2_pca_autoencoder.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PCA is equivalent to Linear auto encoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction\n",
    "Auto Encoder is a popular method in Deep learning. As discussed earlier, PCA is a dimensionality reduction technique used for noise reduction. It is proven that PCA is eqivalent to Linear auto encoder. In PCA the subspace spanned by Principal components is the same subspace spanned by Auto encoder which use linear activation function, and Squared loss function. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Proof : How PCA equivalent to Linear Auto encoder with Squared loss Function ? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case \n",
    "\n",
    "In case of PCA, minimum reconstruction error is defined as : \n",
    "\n",
    "$\\mathcal{L}(\\mathbf{x}, \\tilde{\\mathbf{x}})=\\|\\mathbf{x}-\\tilde{\\mathbf{x}}\\|^{2}$\n",
    "\n",
    "<img src=\"images/1.png\"\n",
    "     alt=\"Markdown Monster icon\"\n",
    "     style=\"float: left; margin-right: 10px;\" />\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "$\\mathbf{z}=f(W_1 \\mathbf{x}) ; \\quad \\hat{\\mathbf{x}}=g(W_2 \\mathbf{z})$\n",
    "\n",
    "\n",
    " Squared loss error is given by :\n",
    " \n",
    " $\\min _{\\mathbf{W_1}, \\mathbf{W_2}} \\frac{1}{2 N} \\sum_{n=1}^{N}\\left\\|\\mathbf{x}^{(n)}-\\hat{\\mathbf{x}}^{(n)}\\right\\|^{2}$\n",
    "\n",
    "\n",
    "If we Assume functions f and g linear i.e..,\n",
    "\n",
    "$\\min _{\\mathbf{W_1}, \\mathbf{W_2}} \\frac{1}{2 N} \\sum_{n=1}^{N}\\left\\|\\mathbf{x}^{(n)}-W_2 W_1 \\mathbf{x}^{(n)}\\right\\|^{2}$\n",
    "\n",
    "If we Consider in Auto encoder we have\n",
    "\n",
    "$\\tilde{\\mathbf{x}}=\\mathbf{W}_{2} \\mathbf{W}_{1} \\mathbf{x}$\n",
    "\n",
    "Under the Constraint,\n",
    "\n",
    "$\\mathbf{W}_{2} \\mathbf{W}_{1} = \\mathbf{I}$ \n",
    "\n",
    "The above Optimisation problem of Auto encoder is equivalent to PCA.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Note : Optimal weights in Auto encoder defined above are equivalent to Principal Components in PCA\n",
    "\n",
    "<img src=\"images/2.png\"\n",
    "     alt=\"Markdown Monster icon\"\n",
    "     style=\"float: left; margin-right: 10px;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Note : Principal components is not equal to Linear Auto encoder weights.\n",
    "\n",
    "### How to recover Principal components from Linear auto Encoder weights ?? \n",
    "\n",
    "\n",
    "### Idea : It is shown that  first m singular vectors of $W_2$ are the first m principal components of X.\n",
    "\n",
    "\n",
    "### \"From Principal Subspaces to Principal Components with Linear Autoencoders\"\"\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}