ch01/First_Insights.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[**Blueprints for Text Analysis Using Python**](https://github.com/blueprints-for-text-analytics-python/blueprints-text)  \n",
    "Jens Albrecht, Sidharth Ramachandran, Christian Winkler\n",
    "\n",
    "**If you like the book or the code examples here, please leave a friendly comment on [Amazon.com](https://www.amazon.com/Blueprints-Text-Analytics-Using-Python/dp/149207408X)!**\n",
    "<img src=\"../rating.png\" width=\"100\"/>\n",
    "\n",
    "\n",
    "# Chapter 1:<div class='tocSkip'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gaining Early Insights from Textual Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Remark<div class='tocSkip'/>\n",
    "\n",
    "The code in this notebook differs slightly from the printed book. For example we frequently use pretty print (`pp.pprint`) instead of `print` and `tqdm`'s `progress_apply` instead of Pandas' `apply`. \n",
    "\n",
    "Moreover, several layout and formatting commands, like `figsize` to control figure size or subplot commands are removed in the book.\n",
    "\n",
    "You may also find some lines marked with three hashes ###. Those are not in the book as well as they don't contribute to the concept.\n",
    "\n",
    "All of this is done to simplify the code in the book and put the focus on the important parts instead of formatting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup<div class='tocSkip'/>\n",
    "\n",
    "Set directory locations. If working on Google Colab: copy files and install required libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:41:58.195850Z",
     "start_time": "2021-05-26T10:41:58.177026Z"
    }
   },
   "outputs": [],
   "source": [
    "import sys, os\n",
    "ON_COLAB = 'google.colab' in sys.modules\n",
    "\n",
    "if ON_COLAB:\n",
    "    GIT_ROOT = 'https://github.com/blueprints-for-text-analytics-python/blueprints-text/raw/master'\n",
    "    os.system(f'wget {GIT_ROOT}/ch01/setup.py')\n",
    "\n",
    "%run -i setup.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Python Settings<div class=\"tocSkip\"/>\n",
    "\n",
    "Common imports, defaults for formatting in Matplotlib, Pandas etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:00.394826Z",
     "start_time": "2021-05-26T10:41:58.197940Z"
    }
   },
   "outputs": [],
   "source": [
    "%run \"$BASE_DIR/settings.py\"\n",
    "\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%config InlineBackend.figure_format = 'png'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What you will learn and what we will build\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploratory Data Analysis\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introducing the Dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:04.546077Z",
     "start_time": "2021-05-26T10:42:00.397654Z"
    }
   },
   "outputs": [],
   "source": [
    "pd.options.display.max_colwidth = 150 ###\n",
    "file = \"un-general-debates-blueprint.csv\"\n",
    "file = f\"{BASE_DIR}/data/un-general-debates/un-general-debates-blueprint.csv.gz\" ### real location\n",
    "df = pd.read_csv(file)\n",
    "df.sample(2, random_state=53)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Blueprint: Getting an Overview of the Data with Pandas\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculating Summary Statistics for Columns\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:04.585120Z",
     "start_time": "2021-05-26T10:42:04.548589Z"
    }
   },
   "outputs": [],
   "source": [
    "df['length'] = df['text'].str.len()\n",
    "\n",
    "df.describe().T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:04.644901Z",
     "start_time": "2021-05-26T10:42:04.587343Z"
    }
   },
   "outputs": [],
   "source": [
    "df[['country', 'speaker']].describe(include='O').T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checking for Missing Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:04.673446Z",
     "start_time": "2021-05-26T10:42:04.646924Z"
    }
   },
   "outputs": [],
   "source": [
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:04.715677Z",
     "start_time": "2021-05-26T10:42:04.679932Z"
    }
   },
   "outputs": [],
   "source": [
    "df['speaker'].fillna('unkown', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:04.752450Z",
     "start_time": "2021-05-26T10:42:04.720245Z"
    }
   },
   "outputs": [],
   "source": [
    "df[df['speaker'].str.contains('Bush')]['speaker'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting Value Distributions\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:04.948204Z",
     "start_time": "2021-05-26T10:42:04.757532Z"
    }
   },
   "outputs": [],
   "source": [
    "df['length'].plot(kind='box', vert=False, figsize=(8, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:05.148991Z",
     "start_time": "2021-05-26T10:42:04.950770Z"
    }
   },
   "outputs": [],
   "source": [
    "df['length'].plot(kind='hist', bins=30, figsize=(8,2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:05.439106Z",
     "start_time": "2021-05-26T10:42:05.151748Z"
    }
   },
   "outputs": [],
   "source": [
    "# Not in book: seaborn plot with gaussian kernel density estimate\n",
    "import seaborn as sns\n",
    "\n",
    "plt.figure(figsize=(8, 2))\n",
    "sns.distplot(df['length'], bins=30, kde=True);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparing Value Distributions across Categories\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:06.102415Z",
     "start_time": "2021-05-26T10:42:05.441397Z"
    }
   },
   "outputs": [],
   "source": [
    "where = df['country'].isin(['USA', 'FRA', 'GBR', 'CHN', 'RUS'])\n",
    "g = sns.catplot(data=df[where], x=\"country\", y=\"length\", kind='box')\n",
    "g.fig.set_size_inches(4, 3) ###\n",
    "g.fig.set_dpi(100) ###\n",
    "g = sns.catplot(data=df[where], x=\"country\", y=\"length\", kind='violin')\n",
    "g.fig.set_size_inches(4, 3) ###\n",
    "g.fig.set_dpi(100) ###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing Developments over Time\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:06.271029Z",
     "start_time": "2021-05-26T10:42:06.104222Z"
    }
   },
   "outputs": [],
   "source": [
    "df.groupby('year').size().plot(title=\"Number of Countries\", figsize=(6,2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:06.558946Z",
     "start_time": "2021-05-26T10:42:06.274077Z"
    }
   },
   "outputs": [],
   "source": [
    "df.groupby('year').agg({'length': 'mean'}) \\\n",
    "  .plot(title=\"Avg. Speech Length\", ylim=(0,30000), figsize=(6,2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Blueprint: Building a Simple Text Preprocessing Pipeline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization with Regular Expressions\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:06.582112Z",
     "start_time": "2021-05-26T10:42:06.562641Z"
    }
   },
   "outputs": [],
   "source": [
    "import regex as re\n",
    "\n",
    "def tokenize(text):\n",
    "    return re.findall(r'[\\w-]*\\p{L}[\\w-]*', text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:06.603608Z",
     "start_time": "2021-05-26T10:42:06.584627Z"
    }
   },
   "outputs": [],
   "source": [
    "text = \"Let's defeat SARS-CoV-2 together in 2020!\"\n",
    "tokens = tokenize(text)\n",
    "print(\"|\".join(tokens))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Treating Stop Words\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:06.788245Z",
     "start_time": "2021-05-26T10:42:06.605812Z"
    }
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "# not in book: make sure stop words are available\n",
    "nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:06.813022Z",
     "start_time": "2021-05-26T10:42:06.792334Z"
    }
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "\n",
    "stopwords = set(nltk.corpus.stopwords.words('english'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:06.835634Z",
     "start_time": "2021-05-26T10:42:06.815532Z"
    }
   },
   "outputs": [],
   "source": [
    "def remove_stop(tokens):\n",
    "    return [t for t in tokens if t.lower() not in stopwords]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:06.857424Z",
     "start_time": "2021-05-26T10:42:06.839934Z"
    }
   },
   "outputs": [],
   "source": [
    "include_stopwords = {'dear', 'regards', 'must', 'would', 'also'}\n",
    "exclude_stopwords = {'against'}\n",
    "\n",
    "stopwords |= include_stopwords\n",
    "stopwords -= exclude_stopwords"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Processing a Pipeline with one Line of Code\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:06.877721Z",
     "start_time": "2021-05-26T10:42:06.859933Z"
    }
   },
   "outputs": [],
   "source": [
    "pipeline = [str.lower, tokenize, remove_stop]\n",
    "\n",
    "def prepare(text, pipeline):\n",
    "    tokens = text\n",
    "    for transform in pipeline:\n",
    "        tokens = transform(tokens)\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:24.578033Z",
     "start_time": "2021-05-26T10:42:06.880896Z"
    }
   },
   "outputs": [],
   "source": [
    "df['tokens'] = df['text'].progress_apply(prepare, pipeline=pipeline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:24.743332Z",
     "start_time": "2021-05-26T10:42:24.587982Z"
    }
   },
   "outputs": [],
   "source": [
    "df['num_tokens'] = df['tokens'].progress_map(len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Blueprints for Word Frequency Analysis\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Blueprint: Counting Words with a Counter\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:24.785986Z",
     "start_time": "2021-05-26T10:42:24.752296Z"
    }
   },
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "tokens = tokenize(\"She likes my cats and my cats like my sofa.\")\n",
    "\n",
    "counter = Counter(tokens)\n",
    "print(counter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:24.818884Z",
     "start_time": "2021-05-26T10:42:24.790881Z"
    }
   },
   "outputs": [],
   "source": [
    "more_tokens = tokenize(\"She likes dogs and cats.\")\n",
    "counter.update(more_tokens)\n",
    "print(counter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:26.700698Z",
     "start_time": "2021-05-26T10:42:24.822554Z"
    }
   },
   "outputs": [],
   "source": [
    "counter = Counter()\n",
    "\n",
    "_ = df['tokens'].map(counter.update)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:26.741941Z",
     "start_time": "2021-05-26T10:42:26.702480Z"
    }
   },
   "outputs": [],
   "source": [
    "pp.pprint(counter.most_common(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:26.765301Z",
     "start_time": "2021-05-26T10:42:26.743907Z"
    }
   },
   "outputs": [],
   "source": [
    "from collections import Counter ###\n",
    "def count_words(df, column='tokens', preprocess=None, min_freq=2):\n",
    "\n",
    "    # process tokens and update counter\n",
    "    def update(doc):\n",
    "        tokens = doc if preprocess is None else preprocess(doc)\n",
    "        counter.update(tokens)\n",
    "\n",
    "    # create counter and run through all data\n",
    "    counter = Counter()\n",
    "    df[column].progress_map(update)\n",
    "\n",
    "    # transform counter into data frame\n",
    "    freq_df = pd.DataFrame.from_dict(counter, orient='index', columns=['freq'])\n",
    "    freq_df = freq_df.query('freq >= @min_freq')\n",
    "    freq_df.index.name = 'token'\n",
    "    \n",
    "    return freq_df.sort_values('freq', ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:28.914276Z",
     "start_time": "2021-05-26T10:42:26.767840Z"
    }
   },
   "outputs": [],
   "source": [
    "freq_df = count_words(df)\n",
    "freq_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:39.172001Z",
     "start_time": "2021-05-26T10:42:28.917074Z"
    }
   },
   "outputs": [],
   "source": [
    "# top words with 10+ characters\n",
    "count_words(df, column='text', \n",
    "            preprocess=lambda text: re.findall(r\"\\w{10,}\", text)).head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Blueprint: Creating a Frequency Diagram\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:39.648788Z",
     "start_time": "2021-05-26T10:42:39.175200Z"
    }
   },
   "outputs": [],
   "source": [
    "ax = freq_df.head(15).plot(kind='barh', width=0.95, figsize=(8,3))\n",
    "ax.invert_yaxis()\n",
    "ax.set(xlabel='Frequency', ylabel='Token', title='Top Words')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Blueprint: Creating Word Clouds\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:40.143556Z",
     "start_time": "2021-05-26T10:42:39.651100Z"
    }
   },
   "outputs": [],
   "source": [
    "from wordcloud import WordCloud\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "text = df.query(\"year==2015 and country=='USA'\")['text'].values[0]\n",
    "\n",
    "plt.figure(figsize=(4, 2)) ###\n",
    "wc = WordCloud(max_words=100, stopwords=stopwords)\n",
    "wc.generate(text)\n",
    "plt.imshow(wc, interpolation='bilinear')\n",
    "plt.axis(\"off\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:40.173113Z",
     "start_time": "2021-05-26T10:42:40.146097Z"
    }
   },
   "outputs": [],
   "source": [
    "from wordcloud import WordCloud ###\n",
    "from collections import Counter ###\n",
    "\n",
    "def wordcloud(word_freq, title=None, max_words=200, stopwords=None):\n",
    "\n",
    "    wc = WordCloud(width=800, height=400, \n",
    "                   background_color= \"black\", colormap=\"Paired\", \n",
    "                   max_font_size=150, max_words=max_words)\n",
    "    \n",
    "    # convert data frame into dict\n",
    "    if type(word_freq) == pd.Series:\n",
    "        counter = Counter(word_freq.fillna(0).to_dict())\n",
    "    else:\n",
    "        counter = word_freq\n",
    "\n",
    "    # filter stop words in frequency counter\n",
    "    if stopwords is not None:\n",
    "        counter = {token:freq for (token, freq) in counter.items() \n",
    "                              if token not in stopwords}\n",
    "    wc.generate_from_frequencies(counter)\n",
    " \n",
    "    plt.title(title) \n",
    "\n",
    "    plt.imshow(wc, interpolation='bilinear')\n",
    "    plt.axis(\"off\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:41.423825Z",
     "start_time": "2021-05-26T10:42:40.175979Z"
    }
   },
   "outputs": [],
   "source": [
    "freq_2015_df = count_words(df[df['year']==2015])\n",
    "plt.figure(figsize=(12,4))\n",
    "plt.subplot(1,2,1)###\n",
    "wordcloud(freq_2015_df['freq'], max_words=100)\n",
    "plt.subplot(1,2,2)###\n",
    "wordcloud(freq_2015_df['freq'], max_words=100, stopwords=freq_df.head(50).index)\n",
    "#plt.tight_layout()###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Blueprint: Ranking with TF-IDF\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:41.457725Z",
     "start_time": "2021-05-26T10:42:41.427473Z"
    }
   },
   "outputs": [],
   "source": [
    "def compute_idf(df, column='tokens', preprocess=None, min_df=2):\n",
    "\n",
    "    def update(doc):\n",
    "        tokens = doc if preprocess is None else preprocess(doc)\n",
    "        counter.update(set(tokens))\n",
    "\n",
    "    # count tokens\n",
    "    counter = Counter()\n",
    "    df[column].progress_map(update)\n",
    "\n",
    "    # create data frame and compute idf\n",
    "    idf_df = pd.DataFrame.from_dict(counter, orient='index', columns=['df'])\n",
    "    idf_df = idf_df.query('df >= @min_df')\n",
    "    idf_df['idf'] = np.log(len(df)/idf_df['df'])+0.1\n",
    "    idf_df.index.name = 'token'\n",
    "    return idf_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:44.140253Z",
     "start_time": "2021-05-26T10:42:41.460618Z"
    }
   },
   "outputs": [],
   "source": [
    "idf_df = compute_idf(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:44.178293Z",
     "start_time": "2021-05-26T10:42:44.145248Z"
    }
   },
   "outputs": [],
   "source": [
    "# Not in book: sample of IDF values\n",
    "# high IDF means rare (interesting) term\n",
    "idf_df.sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:44.414313Z",
     "start_time": "2021-05-26T10:42:44.180732Z"
    }
   },
   "outputs": [],
   "source": [
    "freq_df['tfidf'] = freq_df['freq'] * idf_df['idf']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:44.464686Z",
     "start_time": "2021-05-26T10:42:44.416567Z"
    }
   },
   "outputs": [],
   "source": [
    "# not in book: for more data: joining is faster\n",
    "freq_df = freq_df.join(idf_df)\n",
    "freq_df['tfidf'] = freq_df['freq'] * freq_df['idf']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:48.388348Z",
     "start_time": "2021-05-26T10:42:44.468397Z"
    }
   },
   "outputs": [],
   "source": [
    "freq_1970 = count_words(df[df['year'] == 1970])\n",
    "freq_2015 = count_words(df[df['year'] == 2015])\n",
    "\n",
    "freq_1970['tfidf'] = freq_1970['freq'] * idf_df['idf']\n",
    "freq_2015['tfidf'] = freq_2015['freq'] * idf_df['idf']\n",
    "\n",
    "plt.figure(figsize=(12,6)) ###\n",
    "#wordcloud(freq_df['freq'], title='All years', subplot=(1,3,1))\n",
    "plt.subplot(2,2,1)###\n",
    "wordcloud(freq_1970['freq'], title='1970 - TF', \n",
    "          stopwords=['twenty-fifth', 'twenty-five'])\n",
    "plt.subplot(2,2,2)###\n",
    "wordcloud(freq_2015['freq'], title='2015 - TF', \n",
    "          stopwords=['seventieth'])\n",
    "plt.subplot(2,2,3)###\n",
    "wordcloud(freq_1970['tfidf'], title='1970 - TF-IDF', \n",
    "          stopwords=['twenty-fifth', 'twenty-five', 'twenty', 'fifth'])\n",
    "plt.subplot(2,2,4)###\n",
    "wordcloud(freq_2015['tfidf'], title='2015 - TF-IDF', \n",
    "          stopwords=['seventieth'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Blueprint: Finding a Keyword in Context (KWIC)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** textacy's API had major changes from version 0.10.1 (as used in the book) to 0.11.  \n",
    "Here, `textacy.text_utils.KWIC` became `textacy.extract.kwic.keyword_in_context` (see [textacy documentation](https://textacy.readthedocs.io/en/latest/api_reference/extract.html#module-textacy.extract.kwic)).\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-04T05:09:08.234120Z",
     "start_time": "2021-06-04T05:09:05.181494Z"
    }
   },
   "outputs": [],
   "source": [
    "import textacy\n",
    "\n",
    "if textacy.__version__ < '0.11': # as in printed book\n",
    "    from textacy.text_utils import KWIC\n",
    "    \n",
    "else: # for textacy 0.11.x\n",
    "    from textacy.extract.kwic import keyword_in_context\n",
    "\n",
    "    def KWIC(*args, **kwargs):\n",
    "        # call keyword_in_context with all params except 'print_only'\n",
    "        return keyword_in_context(*args, \n",
    "                           **{kw: arg for kw, arg in kwargs.items() \n",
    "                            if kw != 'print_only'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:58:00.290940Z",
     "start_time": "2021-05-26T10:58:00.270371Z"
    }
   },
   "outputs": [],
   "source": [
    "def kwic(doc_series, keyword, window=35, print_samples=5):\n",
    "\n",
    "    def add_kwic(text):\n",
    "        kwic_list.extend(KWIC(text, keyword, ignore_case=True, \n",
    "                              window_width=window, print_only=False))\n",
    "\n",
    "    kwic_list = []\n",
    "    doc_series.progress_map(add_kwic)\n",
    "\n",
    "    if print_samples is None or print_samples==0:\n",
    "        return kwic_list\n",
    "    else:\n",
    "        k = min(print_samples, len(kwic_list))\n",
    "        print(f\"{k} random samples out of {len(kwic_list)} \" + \\\n",
    "              f\"contexts for '{keyword}':\")\n",
    "        for sample in random.sample(list(kwic_list), k):\n",
    "            print(re.sub(r'[\\n\\t]', ' ', sample[0])+'  '+ \\\n",
    "                  sample[1]+'  '+\\\n",
    "                  re.sub(r'[\\n\\t]', ' ', sample[2]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:58:04.386669Z",
     "start_time": "2021-05-26T10:58:04.296378Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "random.seed(22) ###\n",
    "kwic(df[df['year'] == 2015]['text'], 'sdgs', print_samples=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Blueprint: Analyzing N-Grams\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:49.206581Z",
     "start_time": "2021-05-26T10:41:58.405Z"
    }
   },
   "outputs": [],
   "source": [
    "def ngrams(tokens, n=2, sep=' '):\n",
    "    return [sep.join(ngram) for ngram in zip(*[tokens[i:] for i in range(n)])]\n",
    "\n",
    "text = \"the visible manifestation of the global climate change\"\n",
    "tokens = tokenize(text)\n",
    "print(\"|\".join(ngrams(tokens, 2)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:49.208551Z",
     "start_time": "2021-05-26T10:41:58.407Z"
    }
   },
   "outputs": [],
   "source": [
    "def ngrams(tokens, n=2, sep=' ', stopwords=set()):\n",
    "    return [sep.join(ngram) for ngram in zip(*[tokens[i:] for i in range(n)])\n",
    "            if len([t for t in ngram if t in stopwords])==0]\n",
    "\n",
    "print(\"Bigrams:\", \"|\".join(ngrams(tokens, 2, stopwords=stopwords)))\n",
    "print(\"Trigrams:\", \"|\".join(ngrams(tokens, 3, stopwords=stopwords)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:49.210075Z",
     "start_time": "2021-05-26T10:41:58.410Z"
    }
   },
   "outputs": [],
   "source": [
    "df['bigrams'] = df['text'].progress_apply(prepare, pipeline=[str.lower, tokenize]) \\\n",
    "                          .progress_apply(ngrams, n=2, stopwords=stopwords)\n",
    "\n",
    "count_words(df, 'bigrams').head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:49.211195Z",
     "start_time": "2021-05-26T10:41:58.413Z"
    }
   },
   "outputs": [],
   "source": [
    "idf_df = compute_idf(df) ### re-initialize to be safe\n",
    "# concatenate existing IDF data frame with bigram IDFs\n",
    "idf_df = pd.concat([idf_df, compute_idf(df, 'bigrams', min_df=10)])\n",
    "\n",
    "freq_df = count_words(df[df['year'] == 2015], 'bigrams')\n",
    "freq_df['tfidf'] = freq_df['freq'] * idf_df['idf']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:49.214118Z",
     "start_time": "2021-05-26T10:41:58.416Z"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,6)) ###\n",
    "plt.subplot(1,2,1) ###\n",
    "wordcloud(freq_df['tfidf'], title='all bigrams', max_words=50)\n",
    "\n",
    "plt.subplot(1,2,2) ###\n",
    "# plt.tight_layout() ###\n",
    "where = freq_df.index.str.contains('climate')\n",
    "wordcloud(freq_df[where]['freq'], title='\"climate\" bigrams', max_words=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Blueprint: Comparing Frequencies across Time-Intervals and Categories\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Frequency Timelines\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:49.216415Z",
     "start_time": "2021-05-26T10:41:58.420Z"
    }
   },
   "outputs": [],
   "source": [
    "def count_keywords(tokens, keywords): \n",
    "    tokens = [t for t in tokens if t in keywords]\n",
    "    counter = Counter(tokens)\n",
    "    return [counter.get(k, 0) for k in keywords]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:49.218645Z",
     "start_time": "2021-05-26T10:41:58.423Z"
    }
   },
   "outputs": [],
   "source": [
    "keywords = ['nuclear', 'terrorism', 'climate', 'freedom']\n",
    "tokens = ['nuclear', 'climate', 'climate', 'freedom', 'climate', 'freedom']\n",
    "\n",
    "print(count_keywords(tokens, keywords))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:49.221158Z",
     "start_time": "2021-05-26T10:41:58.426Z"
    }
   },
   "outputs": [],
   "source": [
    "def count_keywords_by(df, by, keywords, column='tokens'):\n",
    "    \n",
    "    df = df.reset_index(drop=True) # if the supplied dataframe has gaps in the index\n",
    "    freq_matrix = df[column].progress_apply(count_keywords, keywords=keywords)\n",
    "    freq_df = pd.DataFrame.from_records(freq_matrix, columns=keywords)\n",
    "    freq_df[by] = df[by] # copy the grouping column(s)\n",
    "    \n",
    "    return freq_df.groupby(by=by).sum().sort_values(by)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:49.223999Z",
     "start_time": "2021-05-26T10:41:58.428Z"
    }
   },
   "outputs": [],
   "source": [
    "freq_df = count_keywords_by(df, by='year', keywords=keywords)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:49.225541Z",
     "start_time": "2021-05-26T10:41:58.430Z"
    }
   },
   "outputs": [],
   "source": [
    "pd.options.display.max_rows = 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:49.226810Z",
     "start_time": "2021-05-26T10:41:58.432Z"
    }
   },
   "outputs": [],
   "source": [
    "pd.options.display.max_rows = 60"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:49.228254Z",
     "start_time": "2021-05-26T10:41:58.434Z"
    }
   },
   "outputs": [],
   "source": [
    "freq_df.plot(kind='line', figsize=(8, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:49.230717Z",
     "start_time": "2021-05-26T10:41:58.437Z"
    }
   },
   "outputs": [],
   "source": [
    "random.seed(23) ###\n",
    "# analyzing mentions of 'climate' before 1980\n",
    "kwic(df.query('year < 1980')['text'], 'climate', window=35, print_samples=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Frequency Heat Maps\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-26T10:42:49.233155Z",
     "start_time": "2021-05-26T10:41:58.441Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "keywords = ['terrorism', 'terrorist', 'nuclear', 'war', 'oil',\n",
    "            'syria', 'syrian', 'refugees', 'migration', 'peacekeeping', \n",
    "            'humanitarian', 'climate', 'change', 'sustainable', 'sdgs']  \n",
    "\n",
    "freq_df = count_keywords_by(df, by='year', keywords=keywords)\n",
    "\n",
    "# compute relative frequencies based on total number of tokens per year\n",
    "freq_df = freq_df.div(df.groupby('year')['num_tokens'].sum(), axis=0)\n",
    "# apply square root as sublinear filter for better contrast\n",
    "freq_df = freq_df.apply(np.sqrt)\n",
    "\n",
    "plt.figure(figsize=(10, 3)) ###\n",
    "sns.set(font_scale=1) ###\n",
    "sns.heatmap(data=freq_df.T, \n",
    "            xticklabels=True, yticklabels=True, cbar=False, cmap=\"Reds\")\n",
    "sns.set(font_scale=1) ###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Closing Remarks\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "281.542px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}