SPAM Filter (Naive Bayes).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SPAM Filter:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll be using sklearn.naive_bayes to train a simple spam classifier! First by loading our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os #This module provides a portable way of using operating system dependent functionality. For our case we are going to use it for handeling paths.\n",
    "import io #manages the input/output (I/O)\n",
    "import numpy as np\n",
    "import string\n",
    "from pandas import DataFrame\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "from bs4 import BeautifulSoup #A python based package for parsing HTML and XML. We'll use it to remove html tags.\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.naive_bayes import MultinomialNB #Fit the the Naive Bayes model\n",
    "import statistics as st"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cleanLine(line):\n",
    "    stopWords=set(stopwords.words('english'))\n",
    "    line=BeautifulSoup(line, \"lxml\").text #removing html tags\n",
    "    line=''.join(el for el in line if el not in set(string.punctuation)) #Removing punctuation\n",
    "    line=line.split()\n",
    "    line=[word for word in line if word not in stopWords]\n",
    "    line=[word.lower() for word in line] #turn to lower case\n",
    "    line=[word for word in line if word.isalpha()] #Remove numeric terms\n",
    "    line=' '.join(line)\n",
    "    return(line)\n",
    "    \n",
    "\n",
    "def readFiles(path):\n",
    "    for root,foldername,filenames in os.walk(path):\n",
    "        for filename in filenames:\n",
    "            path = os.path.join(root,filename)\n",
    "            #Since we don't want to take in consideration the header of th email we will try to skip it and only extract\n",
    "            #the body of the email. The header and body are separated in all the email samples by a blank line.\n",
    "            testBody = False\n",
    "            lines = []\n",
    "            file = io.open(path, 'r', encoding='latin1')\n",
    "            for line in file:\n",
    "                if testBody:\n",
    "                    lines.append(cleanLine(line))\n",
    "                elif line == '\\n':\n",
    "                    testBody = True\n",
    "            file.close()\n",
    "            message = ' '.join(lines)\n",
    "            yield(path, message) #This a cool tech because the sequences created by multiple yield calls are iterable once\n",
    "                                 #and not saved on the memory. So looking at the size of data extacted this is gonna be helpful!\n",
    "\n",
    "\n",
    "# Our main objective is to create a dataframe with its index the path for the email, and 2 columns one containing the \n",
    "#classification (SPAM/HAM) and the last column containing the cleaned body of the email text.\n",
    "\n",
    "\n",
    "def dataFrameFromDirectory(path, classification):\n",
    "    rows = []\n",
    "    index = []\n",
    "    for filename, message in readFiles(path):\n",
    "        rows.append({'message': message, 'class': classification})\n",
    "        index.append(filename)\n",
    "\n",
    "    return DataFrame(rows, index=index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = DataFrame({'message': [], 'class': []}) #Initial cleaned dataset\n",
    "\n",
    "data = data.append(dataFrameFromDirectory(r'C:\\Users\\YsfEss\\Desktop\\emails\\spam', 'spam'))\n",
    "data = data.append(dataFrameFromDirectory(r'C:\\Users\\YsfEss\\Desktop\\emails\\ham', 'ham'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at that DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>message</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>C:\\Users\\YsfEss\\Desktop\\emails\\spam\\00001.7848dde101aa985090474a91ec93fcf0</th>\n",
       "      <td>spam</td>\n",
       "      <td>ype     black display none            tr go...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C:\\Users\\YsfEss\\Desktop\\emails\\spam\\00002.d94f1b97e48ed3b553b3508d116e6a09</th>\n",
       "      <td>spam</td>\n",
       "      <td>fight the risk cancer   slim down guaranteed l...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C:\\Users\\YsfEss\\Desktop\\emails\\spam\\00003.2ee33bc6eacdb11f38d052c44819ba6c</th>\n",
       "      <td>spam</td>\n",
       "      <td>fight the risk cancer   slim down guaranteed l...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C:\\Users\\YsfEss\\Desktop\\emails\\spam\\00004.eac8de8d759b7e74154f142194282724</th>\n",
       "      <td>spam</td>\n",
       "      <td>adult club offers free membership    instant...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C:\\Users\\YsfEss\\Desktop\\emails\\spam\\00005.57696a39d7d84318ce497886896bf90d</th>\n",
       "      <td>spam</td>\n",
       "      <td>i thought might like slim down guaranteed lose...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   class  \\\n",
       "C:\\Users\\YsfEss\\Desktop\\emails\\spam\\00001.7848d...  spam   \n",
       "C:\\Users\\YsfEss\\Desktop\\emails\\spam\\00002.d94f1...  spam   \n",
       "C:\\Users\\YsfEss\\Desktop\\emails\\spam\\00003.2ee33...  spam   \n",
       "C:\\Users\\YsfEss\\Desktop\\emails\\spam\\00004.eac8d...  spam   \n",
       "C:\\Users\\YsfEss\\Desktop\\emails\\spam\\00005.57696...  spam   \n",
       "\n",
       "                                                                                              message  \n",
       "C:\\Users\\YsfEss\\Desktop\\emails\\spam\\00001.7848d...     ype     black display none            tr go...  \n",
       "C:\\Users\\YsfEss\\Desktop\\emails\\spam\\00002.d94f1...  fight the risk cancer   slim down guaranteed l...  \n",
       "C:\\Users\\YsfEss\\Desktop\\emails\\spam\\00003.2ee33...  fight the risk cancer   slim down guaranteed l...  \n",
       "C:\\Users\\YsfEss\\Desktop\\emails\\spam\\00004.eac8d...    adult club offers free membership    instant...  \n",
       "C:\\Users\\YsfEss\\Desktop\\emails\\spam\\00005.57696...  i thought might like slim down guaranteed lose...  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay now time for spliting data into train/test datasets. We'll do a classic random 70/30 split. Idealy, a k-fold cross validation is better, but we'll use this split for this instance. Keep in mind that we have 3000 oservations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "#First let's shuffle the data frame to garanty randomness of the split.\n",
    "shuffledData=data.sample(frac=1) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "testData=shuffledData[0:900]\n",
    "trainData=shuffledData[900:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To keep the model and the 2 phases of training and test unbiased, we must assure that the proportion of 'spam' in the two sets is almost the same as in the full dataset. The percentage of spams in the full dataset is: 16%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Proportion of spams in the train dataset:\n",
      "0.1680952380952381\n",
      "Proportion of spams in the test dataset:\n",
      "0.16333333333333333\n"
     ]
    }
   ],
   "source": [
    "print('Proportion of spams in the train dataset:')\n",
    "print(trainData['class'].tolist().count('spam')/2100)\n",
    "print('Proportion of spams in the test dataset:')\n",
    "print(testData['class'].tolist().count('spam')/900)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, they are pretty close we can move on knowing we didn't screw anything in the sampling phase."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can do the creation of most frequent words vector manually and use it as our feature vector. But I choose to use all words used in any of the emails regardless of its frequency (rather than choosing the words with highest frequencies).\n",
    "Now I will exploit the power of Scikit-learn and I will use a CountVectorizer which allows to turn a collection of text documents into a matrix of token (word in this case) counts.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "27799"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer = CountVectorizer()\n",
    "counts = vectorizer.fit_transform(trainData['message'].values)\n",
    "len(vectorizer.get_feature_names()) #We will print the number of features taken by CountVectorizer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now again we will use a function of Scikit-learn function to fit a Naive Bayes model to the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier = MultinomialNB()\n",
    "labels = trainData['class'].values\n",
    "classifier.fit(counts, labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see the training accuracy attained by this model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions=classifier.predict(counts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on the training dataset is: 99.57142857142857 %\n"
     ]
    }
   ],
   "source": [
    "print('Accuracy on the training dataset is:',100*(predictions==labels).tolist().count(True)/len(predictions),'%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, this not really the measure to be excited over, the right measure that reflects the performance of this algorithm is the the training dataset so let's do that!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on the test dataset is: 98.66666666666667 %\n"
     ]
    }
   ],
   "source": [
    "test_counts = vectorizer.transform(testData['message'].values)\n",
    "testpredictions = classifier.predict(test_counts)\n",
    "testlabels=testData['class'].values\n",
    "print('Accuracy on the test dataset is:',100*(testpredictions==testlabels).tolist().count(True)/len(testpredictions),'%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! We have a good accuracy of 98.66% over the test dataset. So we can judge the filter as a good one.\n",
    "\n",
    "We can further this study, by studying the confusion matrix to investigate what kind of mistakes our model does, since the dataset is unblanced (Only 17% of the emails are actually spam), even the null model would reach an accuracy of 83%! \n",
    "\n",
    "But, since in this situation we would rather let some spam mails pass rather than block a misclassified innocent and maybe important email, we are not very 'obsessed' with minimizing the false negatives (which would be large since the positives are minoritary), since it doesn't hurt the purpose of the classifier priorities explained before. **Warning:** I must underline that this is not absolute, a very high false negative rate and we found ourself with useless a SPAM classifier, but while the global error rate is very low we can get away with this.\n",
    "\n",
    "In other words, the bias due to the nature of the dataset is not very dangerous in this case due to the tendencies of most ML algorithms to favorize the dominant class which is luckily convenient for this problem. \n",
    "\n",
    "Note that this is not always the case, for example, problems having as minority class whether a patient has cancer or not, or whether a bank transaction is fraudulent or not, need furthur studies on the confusion matrix and plotting ROC curves."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}