Python_Text_Analysis.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "e5c70223",
      "metadata": {
        "id": "e5c70223"
      },
      "source": [
        "# Python for Text Analysis\n",
        "This code was developed with help from the following online resources:\n",
        "1. [Word2Vec, by Sankalp Kolhe](https://github.com/sankyfox/word2vec)\n",
        "2. [Place context analysis using Natural Language Processing, by Bo Zhao](https://github.com/jebowe3/geog595/blob/master/06_ai/pe.md)\n",
        "3. [Google News and Leo Tolstoy: Visualizing Word2Vec Word Embeddings using t-SNE, by Sergey Smetanin](https://towardsdatascience.com/google-news-and-leo-tolstoy-visualizing-word2vec-word-embeddings-with-t-sne-11558d8bd4d)\n",
        "\n",
        "First, we need to install the necessary Python libraries. This should take about 6 minutes..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1f98c3bd",
      "metadata": {
        "id": "1f98c3bd"
      },
      "outputs": [],
      "source": [
        "# Install needed python libraries\n",
        "import sys\n",
        "!pip install --upgrade gensim\n",
        "!pip install spacy\n",
        "!spacy download xx_ent_wiki_sm\n",
        "!pip install nltk\n",
        "!pip install wordcloud\n",
        "!pip install geonamescache\n",
        "!pip install geocoder\n",
        "!pip install stop-words\n",
        "!pip install networkx\n",
        "!pip install python-levenshtein\n",
        "!pip install scikit-learn"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "37000001",
      "metadata": {
        "id": "37000001"
      },
      "source": [
        "### Save The Adventures of Sherlock Holmes as a txt File\n",
        "Next, we will import a few Python modules to create a subdirectory called \"txt\" in our project folder where we will save The Adventures of Sherlock Holmes, downloaded from [Gutenberg](https://www.gutenberg.org/files/1661/1661-0.txt). We will call this file \"advsherlock.txt\" and use this for later analysis."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7b0f84c8",
      "metadata": {
        "id": "7b0f84c8"
      },
      "outputs": [],
      "source": [
        "# Import modules\n",
        "import os\n",
        "import requests\n",
        "import nltk\n",
        "from nltk.tag import pos_tag\n",
        "from nltk.stem import WordNetLemmatizer\n",
        "from nltk.tokenize import word_tokenize\n",
        "import glob\n",
        "import codecs\n",
        "\n",
        "# Download punkt from nltk\n",
        "nltk.download('punkt')\n",
        "\n",
        "# Make a subdirectory called \"txt\" in your project folder\n",
        "os.mkdir('txt')\n",
        "\n",
        "# First, download The Adventures of Sherlock Holmes here: https://www.gutenberg.org/files/1661/1661-0.txt\n",
        "url = 'https://www.gutenberg.org/files/1661/1661-0.txt'\n",
        "r = requests.get(url, allow_redirects=True)\n",
        "\n",
        "# Then, save in the subdirectory called \"txt\" in your project folder\n",
        "open('txt/advsherlock.txt', 'wb').write(r.content)\n",
        "\n",
        "# Finally, identify the text file \n",
        "raw_data_files = sorted(glob.glob(\"txt/*.txt\"))\n",
        "\n",
        "# Read the txt file and measure it by characters\n",
        "raw_corpus_1 = u\"\"\n",
        "for file_name in raw_data_files:\n",
        "    print(\"Reading '{0}' ...\".format(file_name))\n",
        "    with codecs.open(file_name,\"r\",\"utf-8\") as f:\n",
        "        raw_corpus_1 += f.read()\n",
        "    print(\"Corpus is now {0} characters long\".format(len(raw_corpus_1)))\n",
        "    print\n",
        "\n",
        "# Print the txt file contents\n",
        "#print(raw_corpus_1)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a0e96614",
      "metadata": {
        "id": "a0e96614"
      },
      "source": [
        "### Remove Unnecessary Text from the txt File\n",
        "Now, we will remove the extra text at the beginning and the end of the file, so that this is not used in the analysis."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "64a70974",
      "metadata": {
        "scrolled": false,
        "id": "64a70974"
      },
      "outputs": [],
      "source": [
        "# Then, remove unnecessary text at the beginning and end\n",
        "raw_corpus_2 = raw_corpus_1.split('XII.   The Adventure of the Copper Beeches')[1]\n",
        "raw_corpus = raw_corpus_2.split('*** END OF THE PROJECT GUTENBERG EBOOK THE ADVENTURES OF SHERLOCK HOLMES ***')[0]\n",
        "\n",
        "# Print the results\n",
        "#print(raw_corpus)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "330794cb",
      "metadata": {
        "id": "330794cb"
      },
      "source": [
        "### Clean, Lemmatize, and Tokenize Text\n",
        "With the following code, we will build a list of stop words that we will remove from analysis. Stop words are common words that are not particularly meaningful, such as \"the\" or \"nevertheless\". Then, we will tokenize the text by sentence. This breaks the text into its component sentences for more effective processing. We will also lemmatize the text, which means that different forms of words (conjugations, declensions, etc.) will be assigned a standardized vocabulary to avoid distinctions between words like \"see\" and \"saw\"."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "35a71056",
      "metadata": {
        "scrolled": false,
        "id": "35a71056"
      },
      "outputs": [],
      "source": [
        "# Clean, Lemmatize, and Tokenize Text\n",
        "\n",
        "# Import modules\n",
        "import re\n",
        "import multiprocessing\n",
        "from nltk.corpus import stopwords\n",
        "from stop_words import get_stop_words\n",
        "\n",
        "# Download nltk stopwords\n",
        "nltk.download('stopwords')\n",
        "\n",
        "# Download packages for the lemmatizer\n",
        "nltk.download('omw-1.4')\n",
        "nltk.download('averaged_perceptron_tagger')\n",
        "nltk.download('wordnet')\n",
        "\n",
        "# Define the stop words that will be removed from the results\n",
        "stop_words = list(get_stop_words('en'))         #Have around 900 stopwords\n",
        "nltk_words = list(stopwords.words('english'))   #Have around 150 stopwords\n",
        "stop_words.extend(nltk_words)\n",
        "\n",
        "# Extend the list with a few extra tangential words\n",
        "stop_words.extend(['upon','well','may','shall','must','might','much','quite','however','away','yet','oh','ah','mr','miss'])\n",
        "\n",
        "# Tokenize sentences in the text\n",
        "tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')\n",
        "raw_sentences = tokenizer.tokenize(raw_corpus)\n",
        "\n",
        "# Define a function to lemmatize the text (consolidate similar words to a standard vocabulary)\n",
        "def lemmatize_all(t):\n",
        "    wnl = WordNetLemmatizer()\n",
        "    for word, tag in pos_tag(word_tokenize(t)):\n",
        "        if tag.startswith(\"NN\"):\n",
        "            yield wnl.lemmatize(word, pos='n')\n",
        "        elif tag.startswith('VB'):\n",
        "            yield wnl.lemmatize(word, pos='v')\n",
        "        elif tag.startswith('JJ'):\n",
        "            yield wnl.lemmatize(word, pos='a')\n",
        "        elif tag.startswith('RB'):\n",
        "            yield wnl.lemmatize(word, pos='r')\n",
        "        else:\n",
        "            yield word\n",
        "\n",
        "# Define a function to remove non-alphabetic characters and stopwords and tokenize words by sentence\n",
        "def sentence_to_wordlist(raw):\n",
        "    clean = re.sub(\"[^a-zA-Z]\",\" \",raw)\n",
        "    words = clean.split()\n",
        "    words = [x.lower() for x in words]\n",
        "    return words\n",
        "\n",
        "# Add these tokens to a \"sentences\" list\n",
        "sentences = []\n",
        "for raw_sentence in raw_sentences:\n",
        "    if len(raw_sentence) > 0:\n",
        "        # Lemmatize each sentence\n",
        "        lemma_sentence = ' '.join(lemmatize_all(raw_sentence))\n",
        "        # Remove non-alphabetic characters, tokenize words, and sent to the list\n",
        "        sentences.append(sentence_to_wordlist(lemma_sentence))\n",
        "\n",
        "# Remove the stopwords and add to a \"new_sentences\" list\n",
        "new_sentences = []\n",
        "for sublist in sentences:\n",
        "    new_sublist = []\n",
        "    for item in sublist:\n",
        "        if item not in stop_words:\n",
        "            new_sublist.append(item)\n",
        "    new_sentences.append(new_sublist)\n",
        "    \n",
        "# Count the tokens\n",
        "token_count = sum([len(s) for s in new_sentences])\n",
        "print(\"The corpus has {0:,} tokens\".format(token_count))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ddaec4e3",
      "metadata": {
        "id": "ddaec4e3"
      },
      "source": [
        "### Build, Train, and Save a Word2Vec Model\n",
        "Here, we will make a Word2Vec model, which will produce word embeddings for our text. This will allow us to determine how similar certain words are to one another within our text, giving us a sense of the context within which important words exist within The Adventures of Sherlock Holmes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "dd952480",
      "metadata": {
        "scrolled": true,
        "id": "dd952480"
      },
      "outputs": [],
      "source": [
        "# Build, Train, and Save a Model\n",
        "\n",
        "import gensim.models.word2vec as w2v\n",
        "\n",
        "# Set parameters for the Word2Vec model\n",
        "num_features = 300 # The number of dimensions (N) of the N-dimensional space that gensim Word2Vec maps the words onto.\n",
        "min_word_count = 5 # Ignores all words with total frequency lower than this.\n",
        "num_workers = 1 # Use these many worker threads to train the model.\n",
        "#num_workers = multiprocessing.cpu_count() # Use these many worker threads to train the model.\n",
        "context_size = 5 # Maximum distance between the current and predicted word within a sentence - 5 words on each side.\n",
        "downsampling = 1e-3 # The threshold for configuring which higher-frequency words are randomly downsampled, useful range is (0, 1e-5).\n",
        "seed = 2\n",
        "epochs = 15 # Number of iterations (epochs) over the corpus.\n",
        "\n",
        "# Define the model with the parameters\n",
        "model = w2v.Word2Vec(sg=1,seed=seed,workers=num_workers,vector_size=num_features,min_count=min_word_count,window=context_size,sample=downsampling,epochs=epochs)\n",
        "\n",
        "# Build the vocab\n",
        "model.build_vocab(new_sentences)\n",
        "\n",
        "print(\"Word2Vec vocabulary length: \",len(model.wv.vectors))\n",
        "\n",
        "#print(sum(1 for _ in new_sentences))\n",
        "#print(sum(1 for _ in new_sentences))\n",
        "#print(sum(1 for _ in new_sentences))\n",
        "\n",
        "# Train the model on the sentences list\n",
        "#model.train(new_sentences, total_examples = token_count, epochs = model.epochs)\n",
        "model.train(new_sentences, total_examples = 4705, epochs = model.epochs)\n",
        "\n",
        "# save the trained file so we can load it anytime\n",
        "if not os.path.exists('model'):\n",
        "    os.mkdir('model')\n",
        "model.save(os.path.join(\"model\",\"model.w2v\"))\n",
        "\n",
        "model = w2v.Word2Vec.load(os.path.join(\"model\",\"model.w2v\"))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6993277b",
      "metadata": {
        "id": "6993277b"
      },
      "source": [
        "### Model Testing with Input Vocabulary\n",
        "In this step, we will test the model we just produced by feeding it a few sample words. You can play around with this and try different words based on random curiosity or something you know about the stories. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f1d1e8f3",
      "metadata": {
        "scrolled": false,
        "id": "f1d1e8f3"
      },
      "outputs": [],
      "source": [
        "# Test the model\n",
        "\n",
        "# Print words most similar to \"watson\"\n",
        "print(\"Watson\", model.wv.most_similar('watson', topn=50))\n",
        "\n",
        "# Print words most similar to \"sherlock\"\n",
        "print(\"Sherlock\", model.wv.most_similar('sherlock', topn=50))\n",
        "\n",
        "# Print the proximity of 'watson', 'mystery', 'irene', and 'solve' to 'sherlock'\n",
        "print(\"Sherlock\", model.wv.distances('sherlock', ('watson', 'mystery', 'irene', 'solve')))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fef6ad22",
      "metadata": {
        "id": "fef6ad22"
      },
      "source": [
        "### Remove Uninteresting Words\n",
        "We want to retrieve the 50 most common words in the text. However, these include quite a few words we may not care to map in our network analysis. I have included the words that I want to exclude from analysis in the lists below, but you may want to keep some of these or exclude other words. This is one part of the process where your subjectivity comes into play. After excluding these words, we will print the new top 50."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "bb9eadcc",
      "metadata": {
        "id": "bb9eadcc"
      },
      "outputs": [],
      "source": [
        "model_vocab = model.wv.index_to_key\n",
        "\n",
        "boring_words = ['one','two','back','yes','never','nothing','right','last','st','every','still','side']\n",
        "boring_verbs = ['say','come','go','take','make','give','leave','get','put']\n",
        "boring_adj = ['little','long','first','round','small']\n",
        "boring_pronouns = ['us']\n",
        "\n",
        "\n",
        "boring_words = boring_words + boring_verbs + boring_adj + boring_pronouns\n",
        "\n",
        "selected_words = []\n",
        "for word in model_vocab:\n",
        "    if word not in boring_words:\n",
        "        selected_words.append(word)\n",
        "\n",
        "print(selected_words[:50])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c10517df",
      "metadata": {
        "id": "c10517df"
      },
      "source": [
        "### Make a Frequency Distribution of the Top 20 Desired Words\n",
        "Now that we have a list of the most common words we want, let's do a quick analysis of each word's frequency of appearance in the text. After running the following code, you should see a graph of the frequency of the top 20 words among our list of 50."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "63d33409",
      "metadata": {
        "scrolled": true,
        "id": "63d33409"
      },
      "outputs": [],
      "source": [
        "from nltk import FreqDist\n",
        "\n",
        "flat_sentences = [item for sublist in new_sentences for item in sublist]\n",
        "\n",
        "top_words = []\n",
        "for word in flat_sentences:\n",
        "    if word in selected_words[:50]:\n",
        "        top_words.append(word)\n",
        "        \n",
        "for word in [top_words]:\n",
        "    fDist = FreqDist(word)\n",
        "    # Produce a frequency plot\n",
        "    fDist.plot(20)\n",
        "    print(fDist.most_common(50))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "959a0e00",
      "metadata": {
        "id": "959a0e00"
      },
      "source": [
        "### Make a Word Cloud of the Top 50 Desired Words\n",
        "Different situations call for different visualizations. A word cloud is a nice, quickly digested, visualization of the relative importance of different words in a text. Larger words appear more frequently than smaller words. Here, we will produce a word cloud of the top 50 desired words, which we can also see with their counts in the list above. Can you get a better sense of the stories from this visualization?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d42849cc",
      "metadata": {
        "id": "d42849cc"
      },
      "outputs": [],
      "source": [
        "# Create a word cloud of the most common (interesting) words\n",
        "from wordcloud import WordCloud\n",
        "\n",
        "print(\"generating wordcloud...\")\n",
        "# Define word cloud parameters\n",
        "wc = WordCloud(background_color=\"white\", max_words=50, prefer_horizontal=1, mask=None, scale=3, stopwords=stop_words, collocations=False)\n",
        "# Generate the word cloud from the frequency distribution\n",
        "wc.generate_from_frequencies(fDist)\n",
        "# Make a subdirectory called \"wordcloud\" in your project folder\n",
        "os.mkdir('wordcloud')\n",
        "# Save the word cloud to a file in the new subdirectory\n",
        "wc.to_file(\"wordcloud/advsherlock-cloud.jpg\")\n",
        "print(\"completed!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c77f96c9",
      "metadata": {
        "id": "c77f96c9"
      },
      "source": [
        "### Isolate, Clean, and Sort Geographical Locations\n",
        "We can use a natural language processing model called [\"xx_ent_wiki_sm\"](https://spacy.io/models/xx) from Spacy to identify all locations in our text. However, the process is imperfect and we need to clean out errors and fictional locations that we cannot map in QGIS. Through the steps below, we will load the model, use it to identify the locations in the text, clean out any non-locational words, check the results against a dictionary of known places, extend this dictionary with other places we know to be true (ex: \"Colony of Victoria\" is now a state in the independent nation of Australia). Then, we will sift our locations through these lists and create a list of fictional locations from any place name that falls through. This \"hother\" list is what we will use in the next step to finalize our clean list of mappable locations from The Adventures of Sherlock Holmes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "43d7f15e",
      "metadata": {
        "scrolled": false,
        "id": "43d7f15e"
      },
      "outputs": [],
      "source": [
        "# Clean text and analyze to isolate locations\n",
        "print(\"importing modules...\")\n",
        "import xx_ent_wiki_sm\n",
        "print(\"importing complete\")\n",
        "\n",
        "print(\"loading language processing model...\")\n",
        "nlp_wk = xx_ent_wiki_sm.load()\n",
        "print(\"loading language processing model complete\")\n",
        "\n",
        "new_text = raw_corpus.replace('\\r\\n',' ').replace('\\r\\n\\r\\n',' ').replace('\\r\\n    ',' ')\n",
        "\n",
        "print(\"running language processing model on text...\")\n",
        "my_doc = nlp_wk(new_text)\n",
        "print(\"running language processing model on text complete\")\n",
        "\n",
        "# these are words that appear in the results that are not places\n",
        "geoStops = ['No', 't’', 'Study', 'Scarlet', 'Germanspeaking', 'Boots', 'House of Ormstein', 'Grand Duke of Cassel', 'stage—', 'Inner Temple', 'Temple', 'Where', 'née Adler', 'City', 'Well,’', 'Londoners', 'Architecture', 'B', 'Dissolved', 'Suburban Bank', 'Vegetarian Restaurant', 'Park', '‘Encyclopaedia,’', 'father—', '’77', 'Hall', 'Assizes', '’82', '’85', 'Colonel Openshaw', 'A', 'C’—', '’83', 'D.D.', 'd’you', 'Lascar', 'Dane', 'Museum—', 'Museum', 'birds—', '‘Which', 'Covent', 'Regency', 'Bengal Artillery', 'Exactly', 'Engineer', '16A', '” That', 'soul.’ He', 'Thumb  How', 'Esq', 'Cal', 'Serpen', 'England—', 'No,’', 'Twice', 'Jephro,’', 'Hers', 'Copper Beeches  Fowler and Miss Rucastle', 'Englishman', 'Upper Swandam Lane', ' ']\n",
        "\n",
        "# the wikipedia natural language processing model can isolate location entities\n",
        "print(\"reading each entity in text for geographical entities...\")\n",
        "# define an empty array\n",
        "geoTxt = []\n",
        "# for each entity in the processed text...\n",
        "for ent in my_doc.ents:\n",
        "    # if its label identifies it as a location...\n",
        "    if ent.label_ == \"LOC\":\n",
        "        # and if it is not in the listed geoStops...\n",
        "        if ent.text not in geoStops:\n",
        "            # append it to the array above\n",
        "            geoTxt.append(ent.text)\n",
        "print(\"geographical entities isolated\")\n",
        "    \n",
        "# importing geonamescache provides us with a dictionary of global locations\n",
        "print(\"sorting geographical entities\")\n",
        "import geonamescache\n",
        "\n",
        "gc = geonamescache.GeonamesCache()\n",
        "\n",
        "# gets nested dictionary for countries\n",
        "countriesdict = gc.get_countries()\n",
        "# gets nested dictionary for us states\n",
        "statesdict = gc.get_us_states()\n",
        "# gets nested dictionary for cities\n",
        "citiesdict = gc.get_cities()\n",
        "\n",
        "def gen_dict_extract(var, key):\n",
        "    if isinstance(var, dict):\n",
        "        for k, v in var.items():\n",
        "            if k == key:\n",
        "                yield v\n",
        "            if isinstance(v, (dict, list)):\n",
        "                yield from gen_dict_extract(v, key)\n",
        "    elif isinstance(var, list):\n",
        "        for d in var:\n",
        "            yield from gen_dict_extract(d, key)\n",
        "\n",
        "# we will define lists of all countries, states, and cities in geonamescache\n",
        "countries = [*gen_dict_extract(countriesdict, 'name')]\n",
        "states = [*gen_dict_extract(statesdict, 'name')]\n",
        "cities = [*gen_dict_extract(citiesdict, 'name')]\n",
        "\n",
        "# we will also create new lists for locations from the text that do not appear in the geonamescache lists - many of these are historical or colloquial\n",
        "hcountries = ['Bohemia', 'England', 'Scotland', 'Great Britain', 'States', 'U.S.A.', 'kingdom of Bohemia']\n",
        "hstates = ['Herefordshire', 'Colony of Victoria', 'Sussex', 'Southern states', 'Carolinas', 'Isle of Wight', 'Middlesex', 'Berkshire', 'Hampshire', 'Oxfordshire', 'Nova Scotia']\n",
        "hcities = ['City of London', 'St. John’s Wood', 'Aldersgate', 'Eton', 'Marseilles', 'Pondicherry', 'Bloomsbury', 'Calcutta', 'Greenwich', 'Carlsbad', '’Frisco', 'Carlsbad. ‘Remarkable']\n",
        "hstreets = ['Pall Mall']\n",
        "hplaces = ['La Scala', 'Scotland Yard', 'America', 'Europe', 'Atlantic', 'Thames', 'Amoy River', 'St. Paul', 'St. Paul’s.’', 'St. James’s Hall I', 'St. James’s Hall', 'St. Saviour’s', 'King’s Cross', 'St. Pancras Hotel', 'Stroud Valley', 'Covent Garden', 'Covent Garden Market', 'Paddington', 'Pacific', 'Hatherley Farm', 'Balmoral', 'St. George’s', 'Westbury House', 'Serpentine', 'Holborn', 'Pentonville', 'Birchmoor', 'Petersfield', 'Charing Cross Station', 'Kensington I', 'London street', 'Saviour’s', 'Rockies', 'Montague Place', 'Black Swan Hotel', 'Charing Cross']\n",
        "hother = ['Coburg Square', 'Saxe-Coburg Square', 'Saxe-Coburg Square. Let', 'Fresno Street', 'Swandam Lane', 'Fordham', 'Eyford', 'Eyford Station', 'Coburg'] # Fictional locations\n",
        "\n",
        "# for all the listed locations...\n",
        "for txt in geoTxt:\n",
        "    # if the location is in any of the geonamescache lists, \n",
        "    # or is a type of street or station or port,\n",
        "    # add it to the new lists above\n",
        "    if txt in countries:\n",
        "        hcountries.append(txt)\n",
        "    if txt in states:\n",
        "        hstates.append(txt)\n",
        "    if txt in cities:\n",
        "        hcities.append(txt)\n",
        "    if 'Street' in txt or 'Square' in txt or 'Road' in txt or 'Gate' in txt or 'Bridge' in txt or 'Lane' in txt or 'Wharf' in txt or 'Avenue' in txt or 'Station' in txt:\n",
        "        hstreets.append(txt)\n",
        "    # otherwise, add it to the \"hother\" list of fictional locations\n",
        "    if txt not in hcountries and txt not in hstates and txt not in hcities and txt not in hstreets and txt not in hplaces:\n",
        "        hother.append(txt)\n",
        "\n",
        "print('done sorting geographical entities')\n",
        "\n",
        "print('items to be removed: ', hother)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1655ca98",
      "metadata": {
        "id": "1655ca98"
      },
      "source": [
        "### Remove the Fictional Locations\n",
        "Using the \"hother\" list, which we can see printed in our results above, we will clean our list of locations by removing the fictional places and non-locations. We will add the locations that do not match the words in this list to a new array called \"locations\". After running this code, you will see the new list of locations printed in the results."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1fed6986",
      "metadata": {
        "scrolled": true,
        "id": "1fed6986"
      },
      "outputs": [],
      "source": [
        "# make a new list without fictional locations\n",
        "print('removing fictional locations and making a new list of locations')\n",
        "# define an empty array\n",
        "locations = []\n",
        "\n",
        "# for all the items in the \"geoTxt\" list...\n",
        "for item in geoTxt:\n",
        "    # if the item is not in the \"hother\" items to be removed list...\n",
        "    if item not in hother:\n",
        "        # append the item to the \"locations\" list\n",
        "        locations.append(item)\n",
        "\n",
        "print('new list complete')\n",
        "print(locations)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "55c15106",
      "metadata": {
        "id": "55c15106"
      },
      "source": [
        "### Create a Listed Frequency Distribution of all Locations\n",
        "Before, we generated a graphed frequency distribution of common words. We could do this again for locations in the text, but let's just make a quick list that will order and count all the locations in our text."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2d366734",
      "metadata": {
        "scrolled": true,
        "id": "2d366734"
      },
      "outputs": [],
      "source": [
        "# Define and print a frequency distribution of geographical entities in the text\n",
        "print('creating a frequency distribution of mentioned locations')\n",
        "frqDist = FreqDist(locations)\n",
        "print(frqDist.most_common(160))\n",
        "print('all done')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fb4f7850",
      "metadata": {
        "id": "fb4f7850"
      },
      "source": [
        "### Create a Graphed Frequency Distribution of the Top 20 Locations\n",
        "Just to demonstrate, let's visualize only the 20 most common mentioned locations in The Adventures of Sherlock Holmes in a frequency plot."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8242e953",
      "metadata": {
        "scrolled": true,
        "id": "8242e953"
      },
      "outputs": [],
      "source": [
        "# Calculate the place frequencies from the resulting list\n",
        "for word in [frqDist]:\n",
        "    lDist = FreqDist(word)\n",
        "    # Produce a frequency plot\n",
        "    lDist.plot(20)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e7268f43",
      "metadata": {
        "id": "e7268f43"
      },
      "source": [
        "### Create a Word Cloud of the Top 50 Locations\n",
        "Just like we did with the most common words, we will make a word cloud of the 50 most-mentioned places in The Adventures of Sherlock Holmes. If you check the results, you will notice that many of the locations are in London, demonstrating how Holmes's sense of place in the world is largely defined by the city in which he lives. However, his geographical knowledge extends to some other surprising places, such as The United States, Australia, and India. Why might this be?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3f4b9e98",
      "metadata": {
        "id": "3f4b9e98"
      },
      "outputs": [],
      "source": [
        "# Create a word cloud of the most common locations\n",
        "from wordcloud import WordCloud\n",
        "\n",
        "print(\"generating wordcloud...\")\n",
        "# Define word cloud parameters\n",
        "wc = WordCloud(background_color=\"white\", max_words=50, prefer_horizontal=1, mask=None, scale=3, stopwords=stop_words, collocations=False)\n",
        "# Generate the word cloud from the frequency distribution\n",
        "wc.generate_from_frequencies(frqDist)\n",
        "# Save the word cloud to a file in the \"wordcloud\" subdirectory\n",
        "wc.to_file(\"wordcloud/advsherlock-loc-cloud.jpg\")\n",
        "print(\"completed!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a4339c7f",
      "metadata": {
        "id": "a4339c7f"
      },
      "source": [
        "### Generate a csv File for the Locations\n",
        "The first step in making sure our locations can be mapped in GIS is to get them into a csv file with each place in its own separate row. We will also add the frequency count for each location so we can weight the results proportionally in QGIS."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e75bfe63",
      "metadata": {
        "id": "e75bfe63"
      },
      "outputs": [],
      "source": [
        "# Create a csv file of the locations\n",
        "print('creating a csv file of the locations...')\n",
        "if not os.path.exists('csv'):\n",
        "    os.mkdir('csv')\n",
        "# write each location to a row in the spreadsheet and add its frequency count as a separate column\n",
        "with open(\"csv/adv-sherlock-places.csv\", \"w\", encoding=\"utf8\") as fp:\n",
        "    for item in frqDist.most_common(160):\n",
        "        try:\n",
        "            fp.write(\"%s, %d\\n\" % (item[0], item[1]))\n",
        "            print(item)\n",
        "        except TypeError as error:\n",
        "            pass\n",
        "print(\"finished!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c274210f",
      "metadata": {
        "id": "c274210f"
      },
      "source": [
        "### Geocode the Locations\n",
        "In this step, we will use a Python module called \"geocoder\" to obtain and add latitude and longitude data in new columns of our csv. At the end of this step, we will have a csv file with name, frequency, lat, and lng column headers with coordinate data for our locations."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b3a64280",
      "metadata": {
        "scrolled": false,
        "id": "b3a64280"
      },
      "outputs": [],
      "source": [
        "# create an initial geocoded csv of locations\n",
        "print('creating an initial geocoded csv of locations...')\n",
        "import geocoder\n",
        "\n",
        "with open(\"csv/adv-sherlock-places-geocoded.csv\", \"w\", encoding=\"utf8\") as geofp:\n",
        "    geofp.write(\"name, frequency, lat, lng\\n\")\n",
        "    with open(\"csv/adv-sherlock-places.csv\", \"r\", encoding=\"utf8\") as fp:\n",
        "        for line in fp.readlines():\n",
        "            location = line.split(\",\")[0]\n",
        "            #freq = int(line.split(\",\")[1])\n",
        "            freq = int(line.split(\",\")[-1])\n",
        "            try:\n",
        "                g = geocoder.arcgis(location)\n",
        "                lat = g.current_result.lat\n",
        "                lng = g.current_result.lng\n",
        "                geofp.write(\"%s, %d, %f, %f\\n\" % (location, freq, lat, lng))\n",
        "                print(location, freq, lat, lng)\n",
        "            except:\n",
        "                pass\n",
        "print(\"finished!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "052970c1",
      "metadata": {
        "id": "052970c1"
      },
      "source": [
        "### Correct and Fine Tune Locational Data\n",
        "An astute observer may notice that some of these coordinates are wrong. Moreover, some of the place names could use a little cleaning. For example, note that Colony of Victoria is listed with a latitude in the Northern Hemisphere. That cannot be Australia! Also, notice the name 'Frisco. This is a colloquialism for San Francisco that the geocoder cannot understand. Therefore, it returns the wrong coordinates. In other cases, such as street or station names, we need to add more specificity. There must be many Metropolitan Stations in the world. We want the one in London, England. If you are doing this step on your own, this is likely to take some time sifting through the results, cross-checking with the text, and looking at the locations in QGIS on a map for final verification. The result of this process is the following corrective code. After running this, you will have a csv that you can use in QGIS to map frequency-weighted locations from the text."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "98f4dd72",
      "metadata": {
        "id": "98f4dd72"
      },
      "outputs": [],
      "source": [
        "# fine tune location data to return corrected coordinates\n",
        "print('revising locations to return corrected coordinates...')\n",
        "revised_locations = locations.copy()\n",
        "\n",
        "for i in range(len(revised_locations)):\n",
        "    if revised_locations[i] == 'Edgeware Road.  “You':\n",
        "        revised_locations[i] = 'Edgware Road London England'\n",
        "    if revised_locations[i] == 'Leadenhall Street. Anybody':\n",
        "        revised_locations[i] = 'Leadenhall Street London'\n",
        "    if revised_locations[i] == 'Leadenhall Street—and—”':\n",
        "        revised_locations[i] = 'Leadenhall Street London'\n",
        "    if revised_locations[i] == 'Baker Street.  “Why':\n",
        "        revised_locations[i] = 'Baker Street'\n",
        "    if revised_locations[i] == 'Baker Street.’ That':\n",
        "        revised_locations[i] = 'Baker Street'\n",
        "    if revised_locations[i] == 'Baker Street?” I':\n",
        "        revised_locations[i] = 'Baker Street'\n",
        "    if revised_locations[i] == 'Baker Street. Nothing':\n",
        "        revised_locations[i] = 'Baker Street'\n",
        "    if revised_locations[i] == 'Baker Street.  But Holmes':\n",
        "        revised_locations[i] = 'Baker Street'\n",
        "    if revised_locations[i] == 'Baker Street. Mrs. Rucastle':\n",
        "        revised_locations[i] = 'Baker Street'              \n",
        "    if revised_locations[i] == 'Brixton Road. My':\n",
        "        revised_locations[i] = 'Brixton Road'        \n",
        "    if revised_locations[i] == 'Lebanon':\n",
        "        revised_locations[i] = 'Lebanon Pennsylvania'\n",
        "    if revised_locations[i] == 'Bohemia':\n",
        "        revised_locations[i] = 'Czech Republic'\n",
        "    if revised_locations[i] == 'kingdom of Bohemia':\n",
        "        revised_locations[i] = 'Czech Republic'\n",
        "    if revised_locations[i] == 'Scotland Yard':\n",
        "        revised_locations[i] = 'Great Scotland Yard'\n",
        "    if revised_locations[i] == 'Reading':\n",
        "        revised_locations[i] = 'Reading England'\n",
        "    if revised_locations[i] == 'Serpentine':\n",
        "        revised_locations[i] = 'The Serpentine London'\n",
        "    if revised_locations[i] == 'Leadenhall Street':\n",
        "        revised_locations[i] = 'Leadenhall Street London'\n",
        "    if revised_locations[i] == 'Surrey':\n",
        "        revised_locations[i] = 'Surrey England'\n",
        "    if revised_locations[i] == 'Streatham':\n",
        "        revised_locations[i] = 'Streatham London'\n",
        "    if revised_locations[i] == 'Kent':\n",
        "        revised_locations[i] = 'Kent England'\n",
        "    if revised_locations[i] == 'Brixton Road':\n",
        "        revised_locations[i] = 'Brixton Road London'\n",
        "    if revised_locations[i] == 'Regent Street':\n",
        "        revised_locations[i] = 'Regent Street London England'\n",
        "    if revised_locations[i] == 'Kensington I':\n",
        "        revised_locations[i] = 'Kensington London England'\n",
        "    if revised_locations[i] == 'Fleet Street':\n",
        "        revised_locations[i] = 'Fleet Street London'\n",
        "    if revised_locations[i] == 'Oxford Street':\n",
        "        revised_locations[i] = 'Oxford Street London'\n",
        "    if revised_locations[i] == 'Paddington Station':\n",
        "        revised_locations[i] = 'Paddington Station London'\n",
        "    if revised_locations[i] == 'Charing Cross':\n",
        "        revised_locations[i] = 'Charing Cross Station London'\n",
        "    if revised_locations[i] == 'States':\n",
        "        revised_locations[i] = 'United States'\n",
        "    if revised_locations[i] == 'Threadneedle Street':\n",
        "        revised_locations[i] = 'Threadneedle Street London'\n",
        "    if revised_locations[i] == 'Bow Street':\n",
        "        revised_locations[i] = 'Bow Street London England'\n",
        "    if revised_locations[i] == 'Paddington':\n",
        "        revised_locations[i] = 'Paddington Station London'\n",
        "    if revised_locations[i] == 'Victoria Street':\n",
        "        revised_locations[i] = 'Queen Victoria Street London England'\n",
        "    if revised_locations[i] == 'Montague Place':\n",
        "        revised_locations[i] = 'Montague Place London England'\n",
        "    if revised_locations[i] == 'St. George’s':        \n",
        "        revised_locations[i] = 'St. George’s Church London England'\n",
        "    if revised_locations[i] == 'Hanover Square':\n",
        "        revised_locations[i] = 'Hanover Square London England'\n",
        "    if revised_locations[i] == 'Lancaster Gate':\n",
        "        revised_locations[i] = 'Lancaster Gate London England'\n",
        "    if revised_locations[i] == 'West End':\n",
        "        revised_locations[i] = 'West End London England'\n",
        "    if revised_locations[i] == 'Carlsbad':\n",
        "        revised_locations[i] = 'Karlovy Vary'\n",
        "    if revised_locations[i] == 'Carlsbad. ‘Remarkable':\n",
        "        revised_locations[i] = 'Karlovy Vary'\n",
        "    if revised_locations[i] == 'La Scala':\n",
        "        revised_locations[i] = 'La Scala Milan'\n",
        "    if revised_locations[i] == 'Serpentine Avenue':\n",
        "        revised_locations[i] = 'Serpentine Avenue London'\n",
        "    if revised_locations[i] == 'St. Paul':\n",
        "        revised_locations[i] = 'St. Paul’s Cathedral London'\n",
        "    if revised_locations[i] == 'St. James’s Hall':\n",
        "        revised_locations[i] = 'St. James’s Hall London England'\n",
        "    if revised_locations[i] == 'St. James’s Hall I':\n",
        "        revised_locations[i] = 'St. James’s Hall London England'\n",
        "    if revised_locations[i] == 'Aldersgate':\n",
        "        revised_locations[i] = 'Aldersgate London England'\n",
        "    if revised_locations[i] == 'Cornwall':\n",
        "        revised_locations[i] = 'Cornwall England'\n",
        "    if revised_locations[i] == 'Farrington Street':\n",
        "        revised_locations[i] = 'Farringdon Street London England'\n",
        "    if revised_locations[i] == 'St. Saviour’s':\n",
        "        revised_locations[i] = 'St. Saviour’s Church Chalk Farm Station London'\n",
        "    if revised_locations[i] == 'Saviour’s':\n",
        "        revised_locations[i] = 'St. Saviour’s Church Chalk Farm Station London'    \n",
        "    if revised_locations[i] == 'Severn':\n",
        "        revised_locations[i] = 'River Severn'\n",
        "    if revised_locations[i] == 'Victoria':\n",
        "        revised_locations[i] = 'Victoria Australia'\n",
        "    if revised_locations[i] == 'Colony of Victoria':\n",
        "        revised_locations[i] = 'Victoria Australia'\n",
        "    if revised_locations[i] == 'Hatherley Farm':\n",
        "        revised_locations[i] = 'Hatherley Farm Gloucester England'\n",
        "    if revised_locations[i] == 'Camberwell':\n",
        "        revised_locations[i] = 'Camberwell London'\n",
        "    if revised_locations[i] == 'Sussex':\n",
        "        revised_locations[i] = 'Sussex England'\n",
        "    if revised_locations[i] == 'East London':\n",
        "        revised_locations[i] = 'East London England'\n",
        "    if revised_locations[i] == 'Carolinas':\n",
        "        revised_locations[i] = 'Hamer South Carolina'\n",
        "    if revised_locations[i] == 'America':\n",
        "        revised_locations[i] = 'United States'\n",
        "    if revised_locations[i] == 'U.S.A.':\n",
        "        revised_locations[i] = 'United States'\n",
        "    if revised_locations[i] == 'Union':\n",
        "        revised_locations[i] = 'United States'\n",
        "    if revised_locations[i] == 'London Bridge':\n",
        "        revised_locations[i] = 'London Bridge England'\n",
        "    if revised_locations[i] == 'Paul’s Wharf':\n",
        "        revised_locations[i] = 'Paul’s Wharf London'\n",
        "    if revised_locations[i] == 'Cannon Street':\n",
        "        revised_locations[i] = 'Cannon Street London'\n",
        "    if revised_locations[i] == 'Middlesex':\n",
        "        revised_locations[i] = 'Middlesex England'\n",
        "    if revised_locations[i] == 'London Road':\n",
        "        revised_locations[i] = 'London Road London'\n",
        "    if revised_locations[i] == 'Wellington Street':\n",
        "        revised_locations[i] = 'Wellington Street London England'\n",
        "    if revised_locations[i] == 'Thames':\n",
        "        revised_locations[i] = 'River Thames England'\n",
        "    if revised_locations[i] == 'Pall Mall':\n",
        "        revised_locations[i] = 'Pall Mall London England'\n",
        "    if revised_locations[i] == 'Amoy River':\n",
        "        revised_locations[i] = 'Amoy China'\n",
        "    if revised_locations[i] == 'Wimpole Street':\n",
        "        revised_locations[i] = 'Wimpole Street London England'\n",
        "    if revised_locations[i] == 'Harley Street':\n",
        "        revised_locations[i] = 'Harley Street London England'\n",
        "    if revised_locations[i] == 'Wigmore Street':\n",
        "        revised_locations[i] = 'Wigmore Street London England'\n",
        "    if revised_locations[i] == 'Bloomsbury':\n",
        "        revised_locations[i] = 'Bloomsbury London England'\n",
        "    if revised_locations[i] == 'Covent Garden':\n",
        "        revised_locations[i] = 'Covent Garden London England'\n",
        "    if revised_locations[i] == 'Covent Garden Market':\n",
        "        revised_locations[i] = 'Covent Garden Market London England'\n",
        "    if revised_locations[i] == 'Greenwich':\n",
        "        revised_locations[i] = 'Greenwich London England'\n",
        "    if revised_locations[i] == 'Grosvenor Square':\n",
        "        revised_locations[i] = 'Grosvenor Square London England'\n",
        "    if revised_locations[i] == 'Pacific':\n",
        "        revised_locations[i] = 'Pacific Slope'\n",
        "    if revised_locations[i] == 'Hyde Park':\n",
        "        revised_locations[i] = 'Hyde Park London England'\n",
        "    if revised_locations[i] == 'Trafalgar  Square':\n",
        "        revised_locations[i] = 'Trafalgar Square London England'\n",
        "    if revised_locations[i] == 'Trafalgar Square':\n",
        "        revised_locations[i] = 'Trafalgar Square London England'\n",
        "    if revised_locations[i] == 'Gordon Square':\n",
        "        revised_locations[i] = 'Gordon Square London England'\n",
        "    if revised_locations[i] == 'Northumberland Avenue':\n",
        "        revised_locations[i] = 'Northumberland Avenue London England'\n",
        "    if revised_locations[i] == 'Metropolitan Station':\n",
        "        revised_locations[i] = 'Metropolitan Station Buildings London England'\n",
        "    if revised_locations[i] == 'Southampton Road':\n",
        "        revised_locations[i] = 'Southampton Road London England'\n",
        "    if revised_locations[i] == 'St. Paul’s.’':\n",
        "        revised_locations[i] = 'St. Paul’s Cathedral London'\n",
        "    if revised_locations[i] == 'Waterloo Bridge':\n",
        "        revised_locations[i] = 'Waterloo Bridge London England'\n",
        "    if revised_locations[i] == 'Waterloo Station':\n",
        "        revised_locations[i] = 'Waterloo Station London England'\n",
        "    if revised_locations[i] == 'High Street':\n",
        "        revised_locations[i] = 'High Street Winchester England'\n",
        "    if revised_locations[i] == 'Black Swan Hotel':\n",
        "        revised_locations[i] = 'High Street Winchester England'\n",
        "    if revised_locations[i] == 'London street':\n",
        "        revised_locations[i] = 'London'\n",
        "    if revised_locations[i] == 'Rockies':\n",
        "        revised_locations[i] = 'Rocky Mountains'\n",
        "    if revised_locations[i] == '’Frisco':\n",
        "        revised_locations[i] = 'San Francisco'\n",
        "\n",
        "# Define a frequency distribution of geographical entities in the text\n",
        "frqDist = FreqDist(revised_locations)\n",
        "\n",
        "# Create a csv file of the locations\n",
        "with open(\"csv/adv-sherlock-places.csv\", \"w\", encoding=\"utf8\") as fp:\n",
        "    for item in frqDist.most_common(160):\n",
        "        try:\n",
        "            fp.write(\"%s, %d\\n\" % (item[0], item[1]))\n",
        "        except TypeError as error:\n",
        "            pass\n",
        "\n",
        "import geocoder\n",
        "\n",
        "with open(\"csv/adv-sherlock-places-geocoded.csv\", \"w\", encoding=\"utf8\") as geofp:\n",
        "    geofp.write(\"name, frequency, lat, lng\\n\")\n",
        "    with open(\"csv/adv-sherlock-places.csv\", \"r\", encoding=\"utf8\") as fp:\n",
        "        for line in fp.readlines():\n",
        "            location = line.split(\",\")[0]\n",
        "            #freq = int(line.split(\",\")[1])\n",
        "            freq = int(line.split(\",\")[-1])\n",
        "            try:\n",
        "                g = geocoder.arcgis(location)\n",
        "                lat = g.current_result.lat\n",
        "                lng = g.current_result.lng\n",
        "                geofp.write(\"%s, %d, %f, %f\\n\" % (location, freq, lat, lng))\n",
        "                print(location, freq, lat, lng)\n",
        "            except:\n",
        "                pass\n",
        "print(\"finished!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3b15cb50",
      "metadata": {
        "id": "3b15cb50"
      },
      "source": [
        "### Create a Gephi File for Network Analysis\n",
        "This code will create a gexf file, which we can open in Gephi to map networks of word associations in The Adventures of Sherlock Holmes. As a preview, the following code takes the top 50 words of interest from our frequency distribution and, for each, prints the 20 most similar words in the text, along with their degree of similarity."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1da86c36",
      "metadata": {
        "id": "1da86c36"
      },
      "outputs": [],
      "source": [
        "import networkx as nx\n",
        "\n",
        "g = nx.DiGraph()\n",
        "items = fDist.most_common(50)\n",
        "for item in items:\n",
        "    g.add_nodes_from(item[0])\n",
        "    try:\n",
        "        mswords = model.wv.most_similar(item[0], topn=20)\n",
        "        for msword in mswords:\n",
        "            g.add_nodes_from(msword[0])\n",
        "            g.add_edge(item[0], msword[0], weight=msword[1])\n",
        "            print(\"%s --> %s: %8.5f\" % (item[0], msword[0], msword[1]))\n",
        "    except KeyError as error:\n",
        "        print(error)\n",
        "\n",
        "# Make a subdirectory called \"gephi\" in your project folder\n",
        "os.mkdir('gephi')\n",
        "\n",
        "# Write and save the gexf file\n",
        "nx.write_gexf(g, \"gephi/adv-sherlock.gexf\", encoding='utf-8', prettyprint=True, version='1.1draft')\n",
        "print(\"finished!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ec79d943",
      "metadata": {
        "id": "ec79d943"
      },
      "source": [
        "### Other Things...\n",
        "The following code blocks are just a few experiments and are not fully fleshed out."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8a1d91d1",
      "metadata": {
        "id": "8a1d91d1"
      },
      "outputs": [],
      "source": [
        "from sklearn.manifold import TSNE\n",
        "import numpy as np\n",
        "\n",
        "# Make a key list from the most frequent 20 words in the model vocabulary\n",
        "#keys = model.wv.index_to_key[:20]\n",
        "\n",
        "# Or pick your own words to test\n",
        "keys = ['sherlock','watson','mystery','train','woman','nature','good','evil']\n",
        "\n",
        "embedding_clusters = []\n",
        "word_clusters = []\n",
        "\n",
        "for word in keys:\n",
        "    embeddings = []\n",
        "    words = []\n",
        "    for similar_word, _ in model.wv.most_similar(word, topn=30):\n",
        "        words.append(similar_word)\n",
        "        embeddings.append(model.wv[similar_word])\n",
        "    embedding_clusters.append(embeddings)\n",
        "    word_clusters.append(words)\n",
        "\n",
        "embedding_clusters = np.array(embedding_clusters)\n",
        "n, m, k = embedding_clusters.shape\n",
        "tsne_model_en_2d = TSNE(perplexity=15, n_components=2, init='pca', n_iter=3500, random_state=32)\n",
        "embeddings_en_2d = np.array(tsne_model_en_2d.fit_transform(embedding_clusters.reshape(n * m, k))).reshape(n, m, 2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1b7035fe",
      "metadata": {
        "scrolled": true,
        "id": "1b7035fe"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.cm as cm\n",
        "%matplotlib inline\n",
        "\n",
        "\n",
        "def tsne_plot_similar_words(title, labels, embedding_clusters, word_clusters, a, filename=None):\n",
        "    plt.figure(figsize=(16, 9))\n",
        "    colors = cm.rainbow(np.linspace(0, 1, len(labels)))\n",
        "    for label, embeddings, words, color in zip(labels, embedding_clusters, word_clusters, colors):\n",
        "        x = embeddings[:, 0]\n",
        "        y = embeddings[:, 1]\n",
        "        plt.scatter(x, y, c=color, alpha=a, label=label)\n",
        "        for i, word in enumerate(words):\n",
        "            plt.annotate(word, alpha=0.5, xy=(x[i], y[i]), xytext=(5, 2),\n",
        "                         textcoords='offset points', ha='right', va='bottom', size=8)\n",
        "    plt.legend(loc=4)\n",
        "    plt.title(title)\n",
        "    plt.grid(True)\n",
        "    if filename:\n",
        "        plt.savefig(filename, format='png', dpi=150, bbox_inches='tight')\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "tsne_plot_similar_words('Similar Words From The Adventures of Sherlock Holmes', keys, embeddings_en_2d, word_clusters, 0.7,\n",
        "                        'similar_words.png')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ed78a2ae",
      "metadata": {
        "id": "ed78a2ae"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.13"
    },
    "colab": {
      "provenance": [],
      "collapsed_sections": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}