01_project_summary.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fake News Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem Description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our social media company has been accused of allowing “fake news” to proliferate on our site.  We are facing a backlash from the public and our shareholders who are pressuring us to respond to the charges. In order to respond, we must first attempt to determine if the charges have merit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I will attempt to build a model, to classify news as fake or not, that can then be used in the analysis of the activity on our site."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fake News is defined as \"Untrue information presented as news. It often has the aim of damaging the reputation of a person or entity, or making money through advertising revenue. [wikipedia](https://en.wikipedia.org/wiki/Fake_news)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Original Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hosted on Kaggle at [https://www.kaggle.com/clmentbisaillon/fake-and-real-news-dataset](https://www.kaggle.com/clmentbisaillon/fake-and-real-news-dataset)\n",
    "\n",
    "Using version 1 of the dataset that was uploaded to Kaggle on 2020-03-26\n",
    "\n",
    "It contains data from 2015 to 2018\n",
    "\n",
    "The dataset and the collection techniques are described in more detail [here](https://www.uvic.ca/engineering/ece/isot/assets/docs/ISOT_Fake_News_Dataset_ReadMe.pdf) \n",
    "\n",
    "    \"This dataset was collected from realworld sources; the truthful articles were obtained by crawling articles from Reuters.com (News website). As for the fake news articles, they were collected from different sources. The fake news articles were collected from unreliable websites that were flagged by Politifact (a fact-checking organization in the USA) and Wikipedia. The dataset contains different types of articles on different topics, however, the majority of articles focus on political and World news topics.\"\n",
    "    \n",
    "The data was originally collected by the University of Victoria ISOT Research Lab and can also be downloaded from their website at this [link](https://www.uvic.ca/engineering/ece/isot/datasets/fake-news/index.php)\n",
    "\n",
    "The following citations are requested by the creators of the dataset:\n",
    "* Ahmed H, Traore I, Saad S. “Detecting opinion spams and fake news using text classification”, Journal of Security and Privacy, Volume 1, Issue 1, Wiley, January/February 2018.\n",
    "* Ahmed H, Traore I, Saad S. (2017) “Detection of Online Fake News Using N-Gram Analysis and Machine Learning Techniques. In: Traore I., Woungang I., Awad A. (eds) Intelligent, Secure, and Dependable Systems in Distributed and Cloud Environments. ISDDC 2017. Lecture Notes in Computer Science, vol 10618. Springer, Cham (pp. 127- 138).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Supplemental Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hosted on Kaggle at [https://www.kaggle.com/sameedhayat/guardian-news-dataset](https://www.kaggle.com/sameedhayat/guardian-news-dataset)\n",
    "\n",
    "Using version 1 of the dataset that was uploaded to Kaggle on 2019-06-02\n",
    "\n",
    "It contains data from 2016 to 2018.\n",
    "\n",
    "The dataset on Kaggle contains news stories from the [Guardian](https://www.theguardian.com/us) on several topics, but the only ones used here are from the politics section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# EDA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data cleaning and initial exploratory data analysis is done in the [01_eda.ipynb](notebooks/01_eda.ipynb) notebook.\n",
    "\n",
    "Data cleaning and filtering of the supplemental dataset is done in the [08_clean_guardian_data.ipynb](notebooks/08_clean_guardian_data.ipynb) notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Pre-Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Text pre-processing is in the [02_text_pre_processing.ipynb](notebooks/02_text_pre_processing.ipynb) notebook.\n",
    "\n",
    "Text pre-processing of the supplemental dataset is in the  [08_clean_guardian_data.ipynb](notebooks/08_clean_guardian_data.ipynb) notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During the EDA process, it was observed that by merely looking at the percentage of capital letters in the news story title, the fake and true news stories have very little overlap.  A heuristic model based upon this separation was applied to the data in the [03_percent_capitals_in_title.ipynb](notebooks/03_percent_capitals_in_title.ipynb) notebook.  While this heuristic could provide 98% accuracy on this dataset, it seems unlikely that it would generalize well and could easily be defeated if it was used as some sort of gatekeeping.  This situation can be likened to the continuous updates that need do be done to a spam filter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next some machine learning models were applied to look for a more generalizable solution. Several iterations of a Naive Bayes Classification model with bag of words (BOW) features were done in the [04_nb_bow.ipynb](notebooks/04_nb_bow.ipynb) notebook, and several iterations of a Random Forest Classifier Model with BOW were done in the [05_rf_bow.ipynb](notebooks/05_rf_bow.ipynb) notebook. Using the selected and normalized data that I found worked the best, I did an additional model using tf/idf instead of bag of words in the [06_rf_tfidf.ipynb](notebooks/06_rf_tfidf.ipynb) notebook and I found that the model using tf/idf performed the best up until this point."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike most situations where the machine learning models don't have the desired accuracy, or precision or recall, etc., in this situation, the style of Fake and True news in the dataset was easy to classify even with a simple bag of words model.  Instead of attempting to increase the accuracy of the models, or try different models, I attempted  to generalize the models while still keeping the accuracy, precision and recall high."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the original dataset only contained \"true\" stories from one source, Reuters, I was worried that although I took steps to not overfit the models to this dataset, it may have overfit to it anyway.  I located supplemental data and tested the model I had selected as best with it.  The [results](notebooks/07_classify_story.ipynb) were barely better than random chance, which was very disappointing.  I then integrated some of this new data into the training dataset, [retrained the model](notebooks/09_rf_tfidf_plus_guardian.ipynb), achieving a little bit lower accuracy, but feeling more confident in the new model. I then checked the new model on the full supplemental dataset and was satisfied with the [results](notebooks/10_classify_story_plus_guardian.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model that I found to be the most generalizable while still returning an accuracy of 89% and F1 scores of 0.89 for both fake and true, was a random forest classifier that only considered words from a stopwords list.  By only looking at the stopwords, the people, places, organizations, dates, jargon, and other situation specific references were removed which would make the news more generalizable because the classifications would not be dependent on those removed elements.  This model can be found in the [09_rf_tfidf_plus_guardian.ipynb](notebooks/09_rf_tfidf_plus_guardian.ipynb) notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While this model has a lower accuracy and F1 scores than the heuristic, it offers a more generalized solution that can be expected to perform well on news stories not in the dataset, because the news stories that have been shared on our social network may have come from different sources or writers than the ones from this dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recommendations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Identify news stories being shared on our social network and test them with the model to see if in fact there is a lot of fake news or just some high profile stories that let to the charge."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Because fake news is a major issue, both to society and as a PR issue for our company, regardless of the results of the tests done on the news stories currently being shared or shared in the recent past, going forward all stories shared on our social network that originate at websites identified as news websites should be classified by our model during the posting process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Use the data collected when the news stories are classified to identify any users that are prolific posters of fake news and consult with the legal department to determine if these actions violate our terms of service."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Future Work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Acquire more labeled news stories to improve the model\n",
    "\n",
    "While this dataset contained approximately 35,000 news stories that were a balanced split on classes, the origin of  the news stories classified as true came from only two sources, Reuters and Guardian, and the origin of the news stories classified as fake is unknown.  Having news stories from additional sources for the true class as well as additional news stories that are fake should create a more robust and generalizable model.  Collecting news stories and properly labeling them true or fake is a time consuming and labor intensive process, which is why it wasn't done already."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Identify the origin of news stories\n",
    "\n",
    "As additional stories are collected and labeled, their URL should also be collected, and to the extent possible, determine the source URL for existing stories in our dataset.  Knowing the source of the news story should be valuable data and can be incorporated into the classification process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Develop a more sophisticated (deep learning) model to use for Fake News detection.  \n",
    "\n",
    "While the model developed here did a good job classifying news stories, with the continual advancements in NLP, such as the recent GPT-3, detection should become harder. For example, if the GPT-3 is told to generate a news story in the style of a writer at the New York Times for instance, it would be hard to detect since our model is not actually checking the accuracy of the news story, just its style."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "learn-env",
   "language": "python",
   "name": "learn-env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}