1-basic-text-mining-concepts.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Basic Text-Mining Concepts with Python\n",
    "\n",
    "---\n",
    "---\n",
    "\n",
    "## Introduction\n",
    "This notebook is an introduction to some basic concepts of text wrangling and natural language processing (NLP) you need for this course. With a familiarity of these topics you will better understand the named entity recognition examples in the following notebooks. There is undoubtedly more to learn and consider for each concept, so consider this as a **brief overview** of:\n",
    "\n",
    "* The text-mining pipeline\n",
    "* Tokenising\n",
    "* Normalising\n",
    "* Stopwords\n",
    "* Word stems and lemmatization\n",
    "* Part-of-speech (POS) tagging\n",
    "* Syntactic dependency parsing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "\n",
    "## The Text-Mining Pipeline: 5 Steps of Text-Mining\n",
    "There is no set way to do text-mining, but typically a workflow will involve steps like these:\n",
    "1. Choosing and collecting text\n",
    "2. Cleaning and preparing text\n",
    "3. Exploring data\n",
    "4. Analysing data\n",
    "5. Presenting results\n",
    "\n",
    "![Text-mining pipeline](assets/pipeline.png)\n",
    "\n",
    "You may go through these steps more than once to refine your data and results, and frequently steps may be merged together. The important thing to realise is that steps 1-2 are critical in ensuring your data is capable of actually addressing the goals of your project. You are likely to spend significant time on cleaning and preparing your text.\n",
    "\n",
    "For our purposes, I have saved a copy of Homer's _Iliad_ for us to work with in the [`data`](data) folder as [`data/iliad-butler-2199-0-prepped.txt`](data/iliad-butler-2199-0-prepped.txt). This copy comes from [Project Gutenberg](http://www.gutenberg.org/): [Homer's *Iliad*, translated by Samuel Butler in 1898](http://www.gutenberg.org/ebooks/2199). Therefore, we will start at step 2, with **cleaning and preparing** this text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "## Loading Data from a File\n",
    "\n",
    "First, we need to open the file containing the _Iliad_ text to play with.\n",
    "\n",
    "### Open and Read Text Files\n",
    "To open and read a text file we use the `open()` function and pass the filepath of the text file as an argument.\n",
    "\n",
    "> I am skipping out some detail here of how exactly opening a file works. If you would like to learn more about opening and reading text files, try this guide [Reading and Writing Files in Python](https://realpython.com/read-write-files-python/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import a module that helps with filepaths\n",
    "from pathlib import Path\n",
    "\n",
    "# Create a filepath for the file\n",
    "text_file = Path('data', 'iliad-butler-2199-0-prepped.txt')\n",
    "\n",
    "# Open the file, read it and store the text with the name `iliad`\n",
    "with open(text_file, encoding=\"utf-8\") as file:\n",
    "    iliad = file.read()\n",
    "\n",
    "iliad[0:200]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "## Tokenising with spaCy\n",
    "\n",
    "Before we can do anything with a text we have to transform it into a form that can be manipulated by a computer. \n",
    "\n",
    "**Tokenising** means splitting a text into its individual elements, such as words, sentences, or symbols. Without this, the computer would just 'see' long strings of characters and have no idea what characters might form meaningful groups.\n",
    "\n",
    "---\n",
    "\n",
    "To do some tokenising we are using [spaCy](https://spacy.io), a free and open-source Natural Language Processing (NLP) package. We will also use this same package for named entity recognition (NER) in the rest of the notebooks in this series, when we'll learn a lot more about it. For now, we will just use some of its basic functions.\n",
    "\n",
    "---\n",
    "\n",
    "First we import and load an English language model provided by spaCy (`en_core_web_sm`) and give it the name `nlp`, ready to do the work on the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import en_core_web_sm\n",
    "nlp = en_core_web_sm.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The book is so long that we might have to process only part of it, depending on how much memory your computer has. To be on the safe side, we will slice the book up to character 400000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iliad_small = iliad[0:400000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we pass the text as an argument to the function `nlp` and spaCy does the rest. spaCy processes the text into a **document** that contains a lot of information about the text.\n",
    "\n",
    "This may take a while as the book is long. Watch until the `In [*]:` to the left of the code has finished and turned into an output number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "document = nlp(iliad_small)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We print out the document text stored in `document.text` just to check that spaCy has correctly parsed it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "document.text[0:500]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The document can be treated like a list of **word tokens**, which we can print out using a list comprehension:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = [word.text for word in document]\n",
    "tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write code here to get just the first 20 tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **EXERCISE**: What problems can you spot this these word tokens?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "spaCy has also split the text into **sentence tokens**. The document stores these sentences in the attribute `sents` and again we can print them out using a list comprehension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences = [sent.text for sent in document.sents]\n",
    "sentences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "## Basic Text Processing\n",
    "When spaCy runs its various text-mining functions it takes care of many things for you. For example:\n",
    "\n",
    "* spaCy understands that titlecase, uppercase and lowercase versions of a word may be the same word.\n",
    "* spaCy realises that generally punctuation is not part of the beginning or end of a word.\n",
    "* spaCy knows that words like \"the\" and \"a\" are not meaningful words in English.\n",
    "\n",
    "Nevetherless, you might not always be using spaCy or another powerful library. Even if you are, it's important to understand how basic text processing works as a foundation for more advanced techniques.\n",
    "\n",
    "### Normalise to Lowercase\n",
    "Normalising all words to lowercase ensures that the same word in different cases can be recognised as the same word.\n",
    "\n",
    "For example, we might want 'Shield', 'shield' and 'SHIELD' to be recognised as the same word. Though, in another case, you may not want the word 'Conservative' to be conflated with the word 'conservative'.\n",
    "\n",
    "How can we lowercase all the tokens in the list of tokens `tokens`? By using the string method `lower()` and a list comprehension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens_lower = [token.lower() for token in tokens]\n",
    "tokens_lower[160:180]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove Puctuation\n",
    "Punctuation such as commas, fullstops and apostrophes can complicate processing a dataset. For example, if punctuation is left in, a word count may not be accurate.\n",
    "\n",
    "This is a complicated matter, however, and what you choose to do would vary depending on your project. It may be appropriate to remove punctuation at different stages of processing. In our case we are going to remove it *after* the text has been tokenised.\n",
    "\n",
    "We will replace *all* punctuation with the empty string `''`. (You do not need to understand this code fully.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import a module that helps with string processing\n",
    "import string\n",
    "\n",
    "# Make a table that translates all punctuation to an empty value (`None`)\n",
    "table = str.maketrans('', '', string.punctuation)\n",
    "punc_table = {chr(key):value for (key, value) in table.items()}\n",
    "\n",
    "# Print the punctuation translation table to inspect it\n",
    "punc_table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the translation\n",
    "tokens_nopunct = [token.translate(table) for token in tokens_lower]\n",
    "tokens_nopunct[160:180]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove Non-Word Tokens\n",
    "\n",
    "We are still left with some problematic tokens that are not useful words, such as empty tokens (`''`) and newline characters (`\\r`, `\\n`).\n",
    "\n",
    "We can try a filter condition for the empty tokens. The operator `!=` is the negative equality operator, so `if token != ''` means \"if token is _not_ equal to the empty string\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens_notempty = [token for token in tokens_nopunct if token != '']\n",
    "tokens_notempty[140:160]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The operator `==` is the equality operator. If you just want a list of empty tokens, write the list comprehension replacing the `!=` with `==`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write code here to get a list of empty tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can remove all the newline characters by adding a condition that filters out all non-alphabetic characters. The string method to use is `isalpha()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = [token for token in tokens_notempty if token.isalpha()]\n",
    "tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### English Stopwords\n",
    "Let's say we are interested in a frequency analysis of words in this book. In other words, we want to find out what are the most common words in order to get an idea about its contents.\n",
    "\n",
    "But not all words are equally interesting. Some common words in English carry little meaning, such as \"the\", \"a\" and \"its\". These are called **stopwords**. There is no definitive list of stopwords, but most Python packages used for Natural Language Processing provide one as a starting point, and spaCy is no exception."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the spaCy standard stopwords list\n",
    "from spacy.lang.en.stop_words import STOP_WORDS\n",
    "stopwords = [stop for stop in STOP_WORDS]\n",
    "\n",
    "# Sort the stopwords in alphabetical order to make them easier to inspect\n",
    "sorted(stopwords)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write code here to count the number of stopwords"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **EXERCISE**: What do you notice about these stopwords?\n",
    "\n",
    "For your own projects you would need to consider which stopwords are most appropriate:\n",
    "* Will standard stopword lists for modern languages be suitable for that language written 10, 50, 200 years ago?\n",
    "* Are there special stopwords specific to the topic or style of literature?\n",
    "* How might you find or create your own stopword list?\n",
    "\n",
    "Now we can filter out the stopwords that match this list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens_nostops = [token for token in tokens if token not in stopwords]\n",
    "tokens_nostops"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a Frequency Distribution\n",
    "Just to demonstrate how nice and clean our tokens are now, we will create a frequency distribution by counting the frequency of each unique word in the text.\n",
    "\n",
    "First, we create a frequency distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import a module that helps with counting\n",
    "from collections import Counter\n",
    "\n",
    "# Count the frequency of words\n",
    "word_freq = Counter(tokens_nostops)\n",
    "word_freq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This `Counter` maps each word to the number of times it appears in the text, e.g. `'coward': 17`. By scrolling down the list you can inspect what look like common and infrequent words.\n",
    "\n",
    "Now we can get precisely the 10 most common words using the function `most_common()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "common_words = word_freq.most_common(10)\n",
    "common_words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualise Results\n",
    "Visualising results can be very useful during text processing to review how well things are going.\n",
    "\n",
    "There are many options for displaying simple charts, and very complex data, in Jupyter notebooks. We are going to use the most well-known library called [Matplotlib](https://matplotlib.org/), although it is perhaps not the easiest to use compared with some others.\n",
    "\n",
    "We don't need to dwell on details of this code as we won't be using Matplotlib again in this course.\n",
    "\n",
    "Let's display our results as a simple line plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display the plot inline in the notebook with interactive controls\n",
    "%matplotlib notebook\n",
    "\n",
    "# Import the matplotlib plot function\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Get a list of the most common words\n",
    "words = [word for word,_ in common_words]\n",
    "\n",
    "# Get a list of the frequency counts for these words\n",
    "freqs = [count for _,count in common_words]\n",
    "\n",
    "# Set titles, labels, ticks and gridlines\n",
    "plt.title(\"Top 10 Words used in Homer's Iliad in English translation\")\n",
    "plt.xlabel(\"Word\")\n",
    "plt.ylabel(\"Count\")\n",
    "plt.xticks(range(len(words)), [str(s) for s in words], rotation=90)\n",
    "plt.grid(b=True, which='major', color='#333333', linestyle='--', alpha=0.2)\n",
    "\n",
    "# Plot the frequency counts\n",
    "plt.plot(freqs)\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "## Word Stems and Lemmatization\n",
    "One form of normalisation we have not yet done is to make sure that different **inflections** of the same word are counted together. In English, words are modified to express quantity, tense, etc. (i.e. **declension** and **conjugation**).\n",
    "\n",
    "For example, 'fish', 'fishes', 'fishy' and 'fishing' are all formed from the root 'fish'.\n",
    "\n",
    "There are two main ways to normalise for inflection:\n",
    "\n",
    "* **Stemming** is reducing a word to a stem by removing endings (a **stem** may not be an actual word).\n",
    "* **Lemmatization** is reducing a word to its meaningful base or dictionary form using its context (a **lemma** is typically a proper word in the language).\n",
    "\n",
    "We will only cover lemmas here.\n",
    "\n",
    "---\n",
    "### Lemmatization with spaCy\n",
    "When we first processed the _Iliad_ text with spaCy [above](#Tokenising-with-spaCy) it created lemmas for all the tokens automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lemmas = [(token.text, token.lemma_) for token in document if token.text.isalpha()]\n",
    "lemmas_interesting = [lemma for lemma in lemmas if lemma[0] != lemma[1] and lemma[1] != '-PRON-']\n",
    "lemmas_interesting[-20:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "## Part-of-Speech (POS) Tagging\n",
    "Another important natural language processing (NLP) task is marking up a word according to its particular **part of speech**. A part of speech is broadly defined as a category of words with similar grammatical properties. In English the following parts of speech are commonly recognised: **noun**, **verb**, **article**, **adjective**, **preposition**, **pronoun**, **adverb**, **conjunction**, and **interjection**.\n",
    "\n",
    "However, in computational linguistics many more sub-categories are recognised.\n",
    "\n",
    "> spaCy follows the [Universal Dependences scheme](https://universaldependencies.org/u/pos/) and a version of the [Penn Treebank tag set](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html). To read about the full set of POS tags see the spaCy documents: [Part-of-speech tagging](https://spacy.io/api/annotation#pos-tagging).\n",
    "\n",
    "Again, spaCy has already POS tagged the text. We just need to look at the document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pos_tags = [(token.text, token.pos_, token.tag_) for token in document if token.text.isalpha()]\n",
    "pos_tags[250:270]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "## Syntactic Dependency Parsing\n",
    "As well as tagging tokens relatively independently from one another, a further form of NLP is tagging tokens according to their context and **relations with other tokens in a sentence**. \n",
    "\n",
    "For example, an adjective (e.g. \"dearest\") of a particular noun (e.g. \"comrades\") might be tagged as being an \"adjectival modifier\" of that _particular_ noun. Parsing a full sentence results in a **tree structure** of how every word in the sentence is related to every other word.\n",
    "\n",
    "> Read more about the syntactic dependency labels used by spaCy in the documentation at [Syntactic Dependency Parsing](https://spacy.io/api/annotation#dependency-parsing).\n",
    "\n",
    "Once more, spaCy has already done this for us. But rather than show you yet another list, this time we can use a nice visualiser called **displaCy** to see this in action.\n",
    "\n",
    "> To play with an online version of syntactic dependency parsing with displaCy see the [displaCy Dependency Visualizer](https://explosion.ai/demos/displacy).\n",
    "\n",
    "To prepare something a little more manageable than the whole _Iliad_ we will take an excerpt and create a new spaCy document first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "excerpt = \"Achilles smiled as he heard this, and was pleased with Antilochus, who was one of his dearest comrades.\"\n",
    "\n",
    "nlp2 = en_core_web_sm.load()\n",
    "document2 = nlp2(excerpt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the displacy package\n",
    "from spacy import displacy\n",
    "\n",
    "# Add some options to display it nicely\n",
    "options = {\"compact\": True, \"distance\": 100, \"color\": \"brown\"}\n",
    "\n",
    "# Pass the excert document to displacy to display\n",
    "displacy.render(document2, options=options)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **EXERCISE**: Try visualizing the parse tree of other sentences and see if you can understand the tags and relations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "## Summary\n",
    "\n",
    "Here's what we have covered in this notebook:\n",
    "\n",
    "* For any project a typical **text-mining pipeline** will involve choosing and collecting text, cleaning and preparing text, exploring data, analysing data and presenting results.\n",
    "* **Tokenising** is splitting a text into its individual elements, such as words, sentences, or symbols.\n",
    "* Basic text processing can include: **normalising** tokens to lowercase, removing punctuation, removing non-word tokens and **stopwords**.\n",
    "* **Lemmatization** is reducing a word to its meaningful base or dictionary form to give its **lemma**.\n",
    "* **Part-of-speech (POS) tagging** is labelling words with the part of speech they represent, for example, noun, verb or adjective.\n",
    "* **Syntactic dependency parsing** is tagging tokens according to their grammatical relations with other tokens in a sentence. Parsing a full sentence results in a **tree structure** of how every word in the sentence is related to every other word.\n",
    "* The NLP library **spaCy** automatically does many of these tasks when you feed it a text to process.\n",
    "\n",
    "In the [next notebook](2-named-entity-recognition-of-henslow-data.ipynb) we will start our case study of another text-mining method: **named entity recognition** with letters from the Henslow Correspondence Project."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}