Classifications.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "** Classifications **\n",
    "\n",
    "The data contains for each app list of tf-idf values which is short for term frequency–inverse document frequency. tf-idf is directly proportionally to the number of times the word appears in the document but inversely proportionally to the number of times the word is present in the corpus. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "%pylab inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "** Load the Pre Processed Data into sparse matrix for memory efficiency **\n",
    "\n",
    "Total Rows = 20104\n",
    "Total Cols = 13626\n",
    "Amount of Data for double = 8 * 20104 * 13626 = 2.09G\n",
    "\n",
    "If we have a limited RAM of 8G we are running out of RAM if we run some classification algorithm on this. So we need to optimize the memory. So storing the data in linked list sparse matrix and then for efficient arithmetic operations converting it to Compressed sparse row matrix. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import csv\n",
    "from scipy.sparse import lil_matrix\n",
    "\n",
    "total_rows = 20104\n",
    "total_cols = 13626\n",
    "\n",
    "app_data = lil_matrix((total_rows, total_cols), dtype='double')\n",
    "app_names_with_labels = []\n",
    "app_name_to_label = dict({})\n",
    "with open('./training/data_with_comma.csv','r') as dest_f:\n",
    "    data_iter = csv.reader(dest_f, \n",
    "                           delimiter = \",\")\n",
    "    \n",
    "    for rowidx, row in enumerate(data_iter):\n",
    "        app_names_with_labels.append((row[0],row[1]))\n",
    "        for colidx, val in zip(*[iter(row[2:])]*2):\n",
    "            app_data[rowidx,colidx] = float(val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Cosine Similarity **\n",
    "\n",
    "If two apps are of similar category most likely they have similar set of words. Like any app which belong to the category of games will have words game in it.\n",
    "\n",
    "We can use cosine similarity to closely match the test set with the highly similar app in the training set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overall Precision = 0.547080129026 Recall 0.539810685965\n",
      "Overall Precision = 0.522065276247 Recall 0.52112201216\n",
      "Overall Precision = 0.556398032519 Recall 0.552126370994\n",
      "Overall Precision = 0.535557038314 Recall 0.536068591557\n",
      "Overall Precision = 0.528987153983 Recall 0.520720674724\n",
      "Overall Precision = 0.541531890467 Recall 0.540921180314\n",
      "Overall Precision = 0.541837802659 Recall 0.537664577136\n",
      "Overall Precision = 0.541692573604 Recall 0.532820614929\n",
      "Overall Precision = 0.536379481561 Recall 0.529833819689\n",
      "Overall Precision = 0.555440131103 Recall 0.553121555571\n"
     ]
    }
   ],
   "source": [
    "\n",
    "import numpy as np\n",
    "import sklearn.metrics.pairwise as smp\n",
    "from sklearn.metrics import precision_recall_fscore_support\n",
    "from sklearn.cross_validation import KFold\n",
    "\n",
    "csr_app_data = app_data.tocsr()\n",
    "app_labels = np.array([ val[1] for val in app_names_with_labels ])\n",
    "\n",
    "kf = KFold(n=20104, n_folds=10)\n",
    "for train_idx, test_idx in kf:\n",
    "    train_data = csr_app_data[train_idx]\n",
    "    train_labels = app_labels[train_idx]\n",
    "    test_data = csr_app_data[test_idx] \n",
    "    test_labels = app_labels[test_idx]\n",
    "    predict_labels = []\n",
    "    for idx, test_vector in enumerate(test_data):\n",
    "            clf = smp.cosine_similarity(train_data, test_vector)\n",
    "            max_idx = clf.argmax()\n",
    "            predict_labels.append(train_labels[max_idx])\n",
    "            \n",
    "    predict_labels_arr = np.array(predict_labels)\n",
    "\n",
    "    (precision, recall, fscore, support) = precision_recall_fscore_support(test_labels, predict_labels_arr)\n",
    "\n",
    "    print(\"Overall Precision = {0} Recall {1}\".format(sum(precision)/precision.size, sum(recall)/recall.size))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Using KNN Algorithm **\n",
    "\n",
    "Tried using the Euclidean distance for the KNN and accuracy seems to be high for 1 Neighbour. May be it is overfitting. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total test data 1057\n",
      "Neighbors: 1, Accuracy: 0.417219\n",
      "Neighbors: 3, Accuracy: 0.343425\n",
      "Neighbors: 5, Accuracy: 0.336802\n",
      "Neighbors: 7, Accuracy: 0.320719\n",
      "Neighbors: 9, Accuracy: 0.324503\n"
     ]
    }
   ],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "import numpy as np\n",
    "csr_app_data = app_data.tocsr()\n",
    "app_labels = np.array([ val[1] for val in app_names_with_labels ])\n",
    "test_idx = np.random.uniform(0, 1, 20104) <= 0.95\n",
    "train_data = csr_app_data[test_idx==True]\n",
    "train_labels = app_labels[test_idx==True]\n",
    "test_data = csr_app_data[test_idx==False] \n",
    "test_labels = app_labels[test_idx==False]\n",
    "print(\"Total test data {0}\".format(len(test_labels)))\n",
    "for n in range(1, 10, 2):\n",
    "    clf = KNeighborsClassifier(n_neighbors=n) \n",
    "    clf.fit(train_data, train_labels)\n",
    "    preds = clf.predict(test_data)\n",
    "    accuracy = np.where(preds==test_labels, 1, 0).sum() / float(len(test_labels))\n",
    "    print \"Neighbors: %d, Accuracy: %3f\" % (n, accuracy)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Predicting with 20,000 training and 104 test\n",
    "\n",
    "The results\n",
    "Neighbors: 1, Accuracy: 0.500000\n",
    "Neighbors: 3, Accuracy: 0.432692\n",
    "Neighbors: 5, Accuracy: 0.451923\n",
    "Neighbors: 7, Accuracy: 0.461538\n",
    "Neighbors: 9, Accuracy: 0.413462\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "** Multinomial Naive Bayes Classification ** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total test data 190\n",
      "Total accuracy 0.631578947368\n"
     ]
    }
   ],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn import linear_model\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "import numpy as np\n",
    "csr_app_data = app_data.tocsr()\n",
    "app_labels = np.array([ val[1] for val in app_names_with_labels ])\n",
    "test_idx = np.random.uniform(0, 1, 20104) <= 0.99\n",
    "train_data = csr_app_data[test_idx==True]\n",
    "train_labels = app_labels[test_idx==True]\n",
    "test_data = csr_app_data[test_idx==False] \n",
    "test_labels = app_labels[test_idx==False]\n",
    "print(\"Total test data {0}\".format(len(test_labels)))\n",
    "clf = MultinomialNB()\n",
    "preds = clf.fit(train_data, train_labels).predict(test_data)\n",
    "accuracy = np.where(preds==test_labels, 1, 0).sum() / float(len(test_labels))\n",
    "print(\"Total accuracy {0}\".format(accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Logistic Regression **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn import linear_model\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "import numpy as np\n",
    "csr_app_data = app_data.tocsr()\n",
    "app_labels = np.array([ val[1] for val in app_names_with_labels ])\n",
    "test_idx = np.random.uniform(0, 1, 20104) <= 0.99\n",
    "train_data = csr_app_data[test_idx==True]\n",
    "train_labels = app_labels[test_idx==True]\n",
    "test_data = csr_app_data[test_idx==False] \n",
    "test_labels = app_labels[test_idx==False]\n",
    "print(\"Total test data {0}\".format(len(test_labels)))\n",
    "clf = linear_model.LogisticRegression(C=1e5)\n",
    "preds = clf.fit(train_data, train_labels).predict(test_data)\n",
    "accuracy = np.where(preds==test_labels, 1, 0).sum() / float(len(test_labels))\n",
    "print(\"Total accuracy {0}\".format(accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Support Vector Machines **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total test data 1959\n",
      "Total accuracy 0.661051556917\n"
     ]
    }
   ],
   "source": [
    "from sklearn import linear_model\n",
    "import numpy as np\n",
    "from sklearn import svm\n",
    "csr_app_data = app_data.tocsr()\n",
    "app_labels = np.array([ val[1] for val in app_names_with_labels ])\n",
    "test_idx = np.random.uniform(0, 1, 20104) <= 0.90\n",
    "train_data = csr_app_data[test_idx==True]\n",
    "train_labels = app_labels[test_idx==True]\n",
    "test_data = csr_app_data[test_idx==False] \n",
    "test_labels = app_labels[test_idx==False]\n",
    "print(\"Total test data {0}\".format(len(test_labels)))\n",
    "clf = svm.LinearSVC()\n",
    "preds = clf.fit(train_data, train_labels).predict(test_data)\n",
    "accuracy = np.where(preds==test_labels, 1, 0).sum() / float(len(test_labels))\n",
    "print(\"Total accuracy {0}\".format(accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total test data 194\n",
      "Total accuracy 0.417525773196\n"
     ]
    }
   ],
   "source": [
    "from sklearn import linear_model\n",
    "import numpy as np\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "csr_app_data = app_data.tocsr()\n",
    "app_labels = np.array([ val[1] for val in app_names_with_labels ])\n",
    "test_idx = np.random.uniform(0, 1, 20104) <= 0.99\n",
    "train_data = csr_app_data[test_idx==True]\n",
    "train_labels = app_labels[test_idx==True]\n",
    "test_data = csr_app_data[test_idx==False] \n",
    "test_labels = app_labels[test_idx==False]\n",
    "print(\"Total test data {0}\".format(len(test_labels)))\n",
    "clf = DecisionTreeClassifier(random_state=0)\n",
    "preds = clf.fit(train_data, train_labels).predict(test_data)\n",
    "accuracy = np.where(preds==test_labels, 1, 0).sum() / float(len(test_labels))\n",
    "print(\"Total accuracy {0}\".format(accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overall Precision = 0.554671136325 Recall 0.552625701271\n",
      "Overall Precision = 0.544333224976 Recall 0.545808353114\n",
      "Overall Precision = 0.553490457444 Recall 0.55381985925\n",
      "Overall Precision = 0.56021611717 Recall 0.557676388879\n",
      "Overall Precision = 0.562168112744 Recall 0.558990193287\n",
      "Overall Precision = 0.5513285504 Recall 0.557144044696\n",
      "Overall Precision = 0.563780594994 Recall 0.565238296047\n",
      "Overall Precision = 0.563819750508 Recall 0.562721201641\n",
      "Overall Precision = 0.566468569446 Recall 0.571718906401\n",
      "Overall Precision = 0.576093341728 Recall 0.582484041026\n"
     ]
    }
   ],
   "source": [
    "from sklearn.decomposition import TruncatedSVD\n",
    "import numpy as np\n",
    "import sklearn.metrics.pairwise as smp\n",
    "from sklearn.metrics import precision_recall_fscore_support\n",
    "from sklearn.cross_validation import KFold\n",
    "\n",
    "csr_app_data = app_data.tocsr()\n",
    "app_labels = np.array([ val[1] for val in app_names_with_labels ])\n",
    "\n",
    "kf = KFold(n=20104, n_folds=10)\n",
    "for train_idx, test_idx in kf:\n",
    "    train_data = csr_app_data[train_idx]\n",
    "    train_labels = app_labels[train_idx]\n",
    "    test_data = csr_app_data[test_idx] \n",
    "    test_labels = app_labels[test_idx]\n",
    "    svd = TruncatedSVD(n_components=200, random_state=42)\n",
    "    reduced_train_data = svd.fit_transform(train_data)\n",
    "    predict_labels = []\n",
    "    for idx, test_vector in enumerate(test_data):\n",
    "        reduced_test_vector = svd.transform(test_vector)\n",
    "        clf = smp.cosine_similarity(reduced_train_data, reduced_test_vector)\n",
    "        max_idx = clf.argmax()\n",
    "        predict_labels.append(train_labels[max_idx])\n",
    "            \n",
    "    predict_labels_arr = np.array(predict_labels)\n",
    "\n",
    "    (precision, recall, fscore, support) = precision_recall_fscore_support(test_labels, predict_labels_arr)\n",
    "\n",
    "    print(\"Overall Precision = {0} Recall {1}\".format(sum(precision)/precision.size, sum(recall)/recall.size))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overall Precision = 0.430767444421 Recall 0.419534708071\n",
      "Overall Precision = 0.420894184762 Recall 0.419155387966\n",
      "Overall Precision = 0.416830715602 Recall 0.421482987822\n",
      "Overall Precision = 0.43380656711 Recall 0.424024198404\n",
      "Overall Precision = 0.424615345529 Recall 0.422128800823\n",
      "Overall Precision = 0.440756297092 Recall 0.439595674659\n",
      "Overall Precision = 0.440507938935 Recall 0.433141657722\n",
      "Overall Precision = 0.435774615425 Recall 0.429264895803\n",
      "Overall Precision = 0.44809568877 Recall 0.446198930497\n",
      "Overall Precision = 0.427699936332 Recall 0.433958775722\n"
     ]
    }
   ],
   "source": [
    "from sklearn.decomposition import TruncatedSVD\n",
    "import numpy as np\n",
    "import sklearn.metrics.pairwise as smp\n",
    "from sklearn.metrics import precision_recall_fscore_support\n",
    "from sklearn.cross_validation import KFold\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "csr_app_data = app_data.tocsr()\n",
    "app_labels = np.array([ val[1] for val in app_names_with_labels ])\n",
    "\n",
    "kf = KFold(n=20104, n_folds=10)\n",
    "for train_idx, test_idx in kf:\n",
    "    train_data = csr_app_data[train_idx]\n",
    "    train_labels = app_labels[train_idx]\n",
    "    test_data = csr_app_data[test_idx] \n",
    "    test_labels = app_labels[test_idx]\n",
    "    svd = TruncatedSVD(n_components=200, random_state=42)\n",
    "    reduced_train_data = svd.fit_transform(train_data)\n",
    "    predict_labels = []\n",
    "    reduced_test_data = svd.transform(test_data)\n",
    "    clf = DecisionTreeClassifier(random_state=42)\n",
    "    preds = clf.fit(reduced_train_data, train_labels).predict(reduced_test_data)\n",
    "\n",
    "    (precision, recall, fscore, support) = precision_recall_fscore_support(test_labels, preds)\n",
    "\n",
    "    print(\"Overall Precision = {0} Recall {1}\".format(sum(precision)/precision.size, sum(recall)/recall.size))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total accuracy 0.636499254102\n",
      "Overall Precision = 0.636397613381 Recall 0.634406484389\n",
      "Total accuracy 0.633515663849\n",
      "Overall Precision = 0.628182638846 Recall 0.629988379777\n",
      "Total accuracy 0.621581302834\n",
      "Overall Precision = 0.619555827851 Recall 0.620463390865\n",
      "Total accuracy 0.633515663849\n",
      "Overall Precision = 0.639366213906 Recall 0.635518105163\n",
      "Total accuracy 0.628855721393\n",
      "Overall Precision = 0.631239262478 Recall 0.623369289271\n",
      "Total accuracy 0.642786069652\n",
      "Overall Precision = 0.624283351653 Recall 0.635799867056\n",
      "Total accuracy 0.642288557214\n",
      "Overall Precision = 0.63849601999 Recall 0.637248098321\n",
      "Total accuracy 0.647263681592\n",
      "Overall Precision = 0.651363791723 Recall 0.636791599146\n",
      "Total accuracy 0.635820895522\n",
      "Overall Precision = 0.637401943493 Recall 0.637253355252\n",
      "Total accuracy 0.63631840796\n",
      "Overall Precision = 0.642025732986 Recall 0.640805784006\n"
     ]
    }
   ],
   "source": [
    "from sklearn.decomposition import TruncatedSVD\n",
    "import numpy as np\n",
    "import sklearn.metrics.pairwise as smp\n",
    "from sklearn.metrics import precision_recall_fscore_support\n",
    "from sklearn.cross_validation import KFold\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "csr_app_data = app_data.tocsr()\n",
    "app_labels = np.array([ val[1] for val in app_names_with_labels ])\n",
    "\n",
    "kf = KFold(n=20104, n_folds=10)\n",
    "for train_idx, test_idx in kf:\n",
    "    train_data = csr_app_data[train_idx]\n",
    "    train_labels = app_labels[train_idx]\n",
    "    test_data = csr_app_data[test_idx] \n",
    "    test_labels = app_labels[test_idx]\n",
    "    svd = TruncatedSVD(n_components=500, random_state=42)\n",
    "    reduced_train_data = svd.fit_transform(train_data)\n",
    "    predict_labels = []\n",
    "    reduced_test_data = svd.transform(test_data)\n",
    "    clf = svm.LinearSVC()\n",
    "    preds = clf.fit(reduced_train_data, train_labels).predict(reduced_test_data)\n",
    "    accuracy = np.where(preds==test_labels, 1, 0).sum() / float(len(test_labels))\n",
    "    print(\"Total accuracy {0}\".format(accuracy))\n",
    "\n",
    "    (precision, recall, fscore, support) = precision_recall_fscore_support(test_labels, preds)\n",
    "\n",
    "    print(\"Overall Precision = {0} Recall {1}\".format(sum(precision)/precision.size, sum(recall)/recall.size))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}