ch08/Topic_Modeling_Clustering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o9qOA_f7ddwG"
   },
   "source": [
    "[**Blueprints for Text Analysis Using Python**](https://github.com/blueprints-for-text-analytics-python/blueprints-text)  \n",
    "Jens Albrecht, Sidharth Ramachandran, Christian Winkler\n",
    "\n",
    "**If you like the book or the code examples here, please leave a friendly comment on [Amazon.com](https://www.amazon.com/Blueprints-Text-Analytics-Using-Python/dp/149207408X)!**\n",
    "<img src=\"https://github.com/blueprints-for-text-analytics-python/blueprints-text/blob/master/rating.png?raw=1\" width=\"100\"/>\n",
    "\n",
    "# Chapter 8: Unsupervised Methods: Topic Modeling and Clustering<div class='tocSkip'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VoQrEGKLddwN"
   },
   "source": [
    "## Remark<div class='tocSkip'/>\n",
    "\n",
    "The code in this notebook differs slightly from the printed book. \n",
    "\n",
    "Several layout and formatting commands, like `figsize` to control figure size or subplot commands are removed in the book.\n",
    "\n",
    "All of this is done to simplify the code in the book and put the focus on the important parts instead of formatting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NHSKkFDjddwO"
   },
   "source": [
    "## Setup<div class='tocSkip'/>\n",
    "\n",
    "Set directory locations. If working on Google Colab: copy files and install required libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "S2kIupv2ddwR",
    "outputId": "955f5151-a959-433f-8dd2-4e4bc6a00be2"
   },
   "outputs": [],
   "source": [
    "import sys, os\n",
    "ON_COLAB = 'google.colab' in sys.modules\n",
    "\n",
    "if ON_COLAB:\n",
    "    GIT_ROOT = 'https://github.com/blueprints-for-text-analytics-python/blueprints-text/raw/master'\n",
    "    os.system(f'wget {GIT_ROOT}/ch08/setup.py')\n",
    "\n",
    "%run -i setup.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o47Xc3PDddwS"
   },
   "source": [
    "## Load Python Settings<div class=\"tocSkip\"/>\n",
    "\n",
    "Common imports, defaults for formatting in Matplotlib, Pandas etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PuQDh8QOddwT"
   },
   "outputs": [],
   "source": [
    "%run \"$BASE_DIR/settings.py\"\n",
    "\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%config InlineBackend.figure_format = 'png'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3To1DXDCddwU"
   },
   "source": [
    "# Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mEpqX1ILddwU",
    "outputId": "00f942d2-37ff-4677-ec6e-4aa5d5065955"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_csv(DEBATES_FILE)\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "WTO3ZGm6ddwU",
    "outputId": "12b22be4-fd3e-4829-8ac6-7402e05bd5de"
   },
   "outputs": [],
   "source": [
    "print(repr(df.iloc[0][\"text\"][0:200]))\n",
    "print(repr(df.iloc[-1][\"text\"][0:200]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0Y5FmLhNddwV"
   },
   "source": [
    "# Split and analyze"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "c_c-RrYcddwW"
   },
   "outputs": [],
   "source": [
    "import re\n",
    "df[\"paragraphs\"] = df[\"text\"].map(lambda text: re.split('\\.\\s*\\n', text))\n",
    "df[\"number_of_paragraphs\"] = df[\"paragraphs\"].map(len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 527
    },
    "id": "FdsLNCg0ddwW",
    "outputId": "c28d0aed-a2d9-418b-a4f1-74b5fada5042",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "df.groupby('year').agg({'number_of_paragraphs': 'mean'}).plot.bar(figsize=(24,12))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qaKScWZ-ddwW"
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from spacy.lang.en.stop_words import STOP_WORDS as stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Iut6ZAeDddwX",
    "outputId": "a1b8d0c7-a5a1-4368-a960-caccf6cb0cbe"
   },
   "outputs": [],
   "source": [
    "tfidf_text_vectorizer = TfidfVectorizer(stop_words=list(stopwords), min_df=5, max_df=0.7)\n",
    "tfidf_text_vectors = tfidf_text_vectorizer.fit_transform(df['text'])\n",
    "tfidf_text_vectors.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "v00R4_9XddwX"
   },
   "outputs": [],
   "source": [
    "# flatten the paragraphs keeping the years\n",
    "paragraph_df = pd.DataFrame([{ \"text\": paragraph, \"year\": year } \n",
    "                               for paragraphs, year in zip(df[\"paragraphs\"], df[\"year\"]) \n",
    "                                    for paragraph in paragraphs if paragraph])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HUMJFJnCddwX",
    "outputId": "c854b718-43e1-443f-ba2f-49b41d9ce278"
   },
   "outputs": [],
   "source": [
    "tfidf_para_vectorizer = TfidfVectorizer(stop_words=list(stopwords), min_df=5, max_df=0.7)\n",
    "tfidf_para_vectors = tfidf_para_vectorizer.fit_transform(paragraph_df[\"text\"])\n",
    "tfidf_para_vectors.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "r476w43YddwX"
   },
   "source": [
    "# NMF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MWZeeaxlddwY"
   },
   "source": [
    "## Speeches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1yvFAwLAddwY"
   },
   "outputs": [],
   "source": [
    "from sklearn.decomposition import NMF\n",
    "\n",
    "nmf_text_model = NMF(n_components=10, random_state=42)\n",
    "W_text_matrix = nmf_text_model.fit_transform(tfidf_text_vectors)\n",
    "H_text_matrix = nmf_text_model.components_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "eyuYMDV3ddwY"
   },
   "outputs": [],
   "source": [
    "def display_topics(model, features, no_top_words=5):\n",
    "    for topic, words in enumerate(model.components_):\n",
    "        total = words.sum()\n",
    "        largest = words.argsort()[::-1] # invert sort order\n",
    "        print(\"\\nTopic %02d\" % topic)\n",
    "        for i in range(0, no_top_words):\n",
    "            print(\"  %s (%2.2f)\" % (features[largest[i]], abs(words[largest[i]]*100.0/total)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZS-bC5DYddwZ",
    "outputId": "c7c3e200-1a27-42da-bdfd-6dc8b7c0f56b"
   },
   "outputs": [],
   "source": [
    "display_topics(nmf_text_model, tfidf_text_vectorizer.get_feature_names_out())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5aC7LQMHddwZ",
    "outputId": "5981f3f3-d0f7-4458-b0c7-4eda85804eee",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "W_text_matrix.sum(axis=0)/W_text_matrix.sum()*100.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8hun3NjCddwa"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rrb-nSh6ddwa"
   },
   "source": [
    "## Paragraphs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rvUbJwidddwa"
   },
   "outputs": [],
   "source": [
    "nmf_para_model = NMF(n_components=10, random_state=42)\n",
    "W_para_matrix = nmf_para_model.fit_transform(tfidf_para_vectors)\n",
    "H_para_matrix = nmf_para_model.components_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bMxxX2erddwa",
    "outputId": "13fd5e41-7cf5-4df0-d6b0-e7c74575bfe7"
   },
   "outputs": [],
   "source": [
    "display_topics(nmf_para_model, tfidf_para_vectorizer.get_feature_names_out())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lXFi24uoddwb",
    "outputId": "1c344d90-8102-4fb0-a006-07c9708b586c"
   },
   "outputs": [],
   "source": [
    "W_para_matrix.sum(axis=0)/W_para_matrix.sum()*100.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5lm8-3lcddwb"
   },
   "source": [
    "# TruncatedSVD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "l3wuY2Sqddwb"
   },
   "outputs": [],
   "source": [
    "from sklearn.decomposition import TruncatedSVD\n",
    "\n",
    "svd_para_model = TruncatedSVD(n_components = 10, random_state=42)\n",
    "W_svd_para_matrix = svd_para_model.fit_transform(tfidf_para_vectors)\n",
    "H_svd_para_matrix = svd_para_model.components_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Mr4-ICyzddwb",
    "outputId": "388847d7-7c2b-4df4-b2a1-3427f25dd3ae"
   },
   "outputs": [],
   "source": [
    "display_topics(svd_para_model, tfidf_para_vectorizer.get_feature_names_out())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ywVQ6Ydhddwb",
    "outputId": "7b19c89d-87e1-450b-d9dc-943018ff359c"
   },
   "outputs": [],
   "source": [
    "svd_para_model.singular_values_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YOAnUrT1ddwb"
   },
   "source": [
    "# LDA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "576bh1FIddwc",
    "outputId": "e305617f-6144-483d-b638-382b61ef17bc"
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "count_para_vectorizer = CountVectorizer(stop_words=list(stopwords), min_df=5, max_df=0.7)\n",
    "count_para_vectors = count_para_vectorizer.fit_transform(paragraph_df[\"text\"])\n",
    "count_para_vectors.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "HNMk0V1eddwc"
   },
   "outputs": [],
   "source": [
    "from sklearn.decomposition import LatentDirichletAllocation\n",
    "\n",
    "lda_para_model = LatentDirichletAllocation(n_components = 10, random_state=42)\n",
    "W_lda_para_matrix = lda_para_model.fit_transform(count_para_vectors)\n",
    "H_lda_para_matrix = lda_para_model.components_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zNNzpGV6ddwc"
   },
   "outputs": [],
   "source": [
    "display_topics(lda_para_model, count_para_vectorizer.get_feature_names_out())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9XEpUa4Zddwc"
   },
   "outputs": [],
   "source": [
    "W_lda_para_matrix.sum(axis=0)/W_lda_para_matrix.sum()*100.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Kc6L1JQlddwd"
   },
   "outputs": [],
   "source": [
    "import pyLDAvis.sklearn\n",
    "\n",
    "lda_display = pyLDAvis.sklearn.prepare(lda_para_model, count_para_vectors, count_para_vectorizer, sort_topics=False)\n",
    "pyLDAvis.display(lda_display)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ul8noVIOddwd"
   },
   "outputs": [],
   "source": [
    "lda_tsne_display = pyLDAvis.sklearn.prepare(lda_para_model, count_para_vectors, count_para_vectorizer, sort_topics=False, mds='tsne')\n",
    "pyLDAvis.display(lda_tsne_display)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "IYHW0BxPddwd"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "naHnuA-Mddw8"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NrUSYuVeddw8"
   },
   "source": [
    "# Display Topic Models and Compare as Wordclouds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Ztm7f2ozddw9"
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from wordcloud import WordCloud\n",
    "\n",
    "def wordcloud_topics(model, features, no_top_words=40):\n",
    "    for topic, words in enumerate(model.components_):\n",
    "        size = {}\n",
    "        largest = words.argsort()[::-1] # invert sort order\n",
    "        for i in range(0, no_top_words):\n",
    "            size[features[largest[i]]] = abs(words[largest[i]])\n",
    "        wc = WordCloud(background_color=\"white\", max_words=100, width=960, height=540)\n",
    "        wc.generate_from_frequencies(size)\n",
    "        plt.figure(figsize=(12,12))\n",
    "        plt.imshow(wc, interpolation='bilinear')\n",
    "        plt.axis(\"off\")\n",
    "        # if you don't want to save the topic model, comment the next line\n",
    "        plt.savefig(f'topic{topic}.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ydTjjKe4ddw9"
   },
   "outputs": [],
   "source": [
    "wordcloud_topics(nmf_para_model, tfidf_para_vectorizer.get_feature_names_out())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "xPgEVLR3ddw-"
   },
   "outputs": [],
   "source": [
    "wordcloud_topics(lda_para_model, count_para_vectorizer.get_feature_names_out())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZY-E5boyddw-"
   },
   "source": [
    "# Time Development of Topic Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zVNAjxSJddw-"
   },
   "outputs": [],
   "source": [
    "W_para_matrix.sum(axis=0)/W_para_matrix.sum()*100.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "--O2iTclddw-"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "before_1990 = np.array(paragraph_df[\"year\"] < 1990)\n",
    "after_1990 = ~ before_1990"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hMBrZaU8ddw_"
   },
   "outputs": [],
   "source": [
    "W_para_matrix_early = nmf_para_model.transform(tfidf_para_vectors[before_1990])\n",
    "W_para_matrix_late  = nmf_para_model.transform(tfidf_para_vectors[after_1990])\n",
    "print(W_para_matrix_early.sum(axis=0)/W_para_matrix_early.sum()*100.0)\n",
    "print(W_para_matrix_late.sum(axis=0)/W_para_matrix_late.sum()*100.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3exKnOqTddw_"
   },
   "outputs": [],
   "source": [
    "['%1.2f'%x for x in W_para_matrix_early.sum(axis=0)/W_para_matrix_early.sum()*100.0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jMgL4F94ddw_"
   },
   "outputs": [],
   "source": [
    "['%1.2f'%x for x in W_para_matrix_early.sum(axis=0)/W_para_matrix_late.sum()*100.0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "nR9iiWpxddxA"
   },
   "outputs": [],
   "source": [
    "from tqdm.auto import tqdm\n",
    "import numpy as np\n",
    "year_data = []\n",
    "for year in tqdm(np.unique(np.unique(paragraph_df[\"year\"]))):\n",
    "    W_year = nmf_para_model.transform(tfidf_para_vectors[np.array(paragraph_df[\"year\"] == year)])\n",
    "    year_data.append([year] + list(W_year.sum(axis=0)/W_year.sum()*100.0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "XQAfrDiCddxA"
   },
   "outputs": [],
   "source": [
    "topic_names = []\n",
    "voc = tfidf_para_vectorizer.get_feature_names_out()\n",
    "for topic in nmf_para_model.components_:\n",
    "    important = topic.argsort()\n",
    "    top_word = voc[important[-1]] + \" \" + voc[important[-2]]\n",
    "    topic_names.append(\"Topic \" + top_word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ilENyfNyddxD"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gWN4bqwnddxD"
   },
   "outputs": [],
   "source": [
    "df_year = pd.DataFrame(year_data, columns=[\"year\"] + topic_names).set_index(\"year\")\n",
    "df_year.plot.area(figsize=(16,9))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mOCo3LhfddxE"
   },
   "outputs": [],
   "source": [
    "[f'Topic {count_para_vectorizer.get_feature_names_out()[words.argsort()[-1]]}' for words in nmf_para_model.components_]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_uPFlMP3ddxE"
   },
   "outputs": [],
   "source": [
    "df_year.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iWg6gfLNddxE"
   },
   "source": [
    "# Optimal number of topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Bu6gicoKddxE"
   },
   "outputs": [],
   "source": [
    "lda_para_model.perplexity(tfidf_para_vectors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "t24A7UeMddxF"
   },
   "outputs": [],
   "source": [
    "from tqdm.auto import tqdm\n",
    "lda_para_model_n = []\n",
    "for n in tqdm(range(5, 21)):\n",
    "    lda_model = LatentDirichletAllocation(n_components = n, random_state=42)\n",
    "    lda_model.fit_transform(count_para_vectors)\n",
    "    lda_perplexity = lda_model.perplexity(tfidf_para_vectors)\n",
    "    lda_para_model_n.append((lda_model, lda_perplexity))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cCJM35u9ddxF"
   },
   "outputs": [],
   "source": [
    "lda_para_model_n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GoStZpqoddxF"
   },
   "source": [
    "# Gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LLJLCAWtddxG"
   },
   "outputs": [],
   "source": [
    "# create tokenized documents\n",
    "gensim_paragraphs = [[w for w in re.findall(r'\\b\\w\\w+\\b' , paragraph.lower()) if w not in stopwords] \n",
    "                           for paragraph in paragraph_df[\"text\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rWwaPq19ddxG"
   },
   "outputs": [],
   "source": [
    "from gensim.corpora import Dictionary\n",
    "\n",
    "dict_gensim_para = Dictionary(gensim_paragraphs) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "s_4gWby7ddxH"
   },
   "outputs": [],
   "source": [
    "dict_gensim_para.filter_extremes(no_below=5, no_above=0.7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QO2NNGZ3ddxH"
   },
   "outputs": [],
   "source": [
    "bow_gensim_para = [dict_gensim_para.doc2bow(paragraph) for paragraph in gensim_paragraphs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2MLO9aR8ddxH"
   },
   "outputs": [],
   "source": [
    "from gensim.models import TfidfModel\n",
    "tfidf_gensim_para = TfidfModel(bow_gensim_para)\n",
    "vectors_gensim_para = tfidf_gensim_para[bow_gensim_para]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7iPOVLH5ddxI"
   },
   "source": [
    "## NMF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tsdgJ966ddxI"
   },
   "outputs": [],
   "source": [
    "from gensim.models.nmf import Nmf\n",
    "\n",
    "nmf_gensim_para = Nmf(vectors_gensim_para, num_topics=10, id2word=dict_gensim_para, kappa=0.1, eval_every=5, random_state=42) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "64E5QhUrddxI"
   },
   "outputs": [],
   "source": [
    "def display_topics_gensim(model):\n",
    "    for topic in range(0, model.num_topics):\n",
    "        print(\"\\nTopic %02d\" % topic)\n",
    "        for (word, prob) in model.show_topic(topic, topn=5):\n",
    "            print(\"  %s (%2.2f)\" % (word, prob))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "h5STJGE1ddxI"
   },
   "outputs": [],
   "source": [
    "display_topics_gensim(nmf_gensim_para)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "EgXOcvShddxI"
   },
   "outputs": [],
   "source": [
    "from gensim.models.coherencemodel import CoherenceModel\n",
    "\n",
    "nmf_gensim_para_coherence = CoherenceModel(model=nmf_gensim_para, texts=gensim_paragraphs, dictionary=dict_gensim_para, coherence='c_v')\n",
    "nmf_gensim_para_coherence_score = nmf_gensim_para_coherence.get_coherence()\n",
    "print(nmf_gensim_para_coherence_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vWBN8awGddxJ"
   },
   "source": [
    "## LDA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7AlxQ68bddxJ"
   },
   "outputs": [],
   "source": [
    "from gensim.models import LdaModel\n",
    "lda_gensim_para = LdaModel(corpus=bow_gensim_para, id2word=dict_gensim_para, chunksize=2000,\n",
    "    alpha='auto', eta='auto', iterations=400, num_topics=10, passes=20, eval_every=None, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zvIIzr-bddxJ"
   },
   "outputs": [],
   "source": [
    "display_topics_gensim(lda_gensim_para)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "sPyuurSFddxJ"
   },
   "outputs": [],
   "source": [
    "lda_gensim_para.log_perplexity(vectors_gensim_para)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hMtiRmzRddxJ"
   },
   "outputs": [],
   "source": [
    "from gensim.models.coherencemodel import CoherenceModel\n",
    "\n",
    "lda_gensim_para_coherence = CoherenceModel(model=lda_gensim_para, texts=gensim_paragraphs, dictionary=dict_gensim_para, coherence='c_v')\n",
    "lda_gensim_para_coherence_score = lda_gensim_para_coherence.get_coherence()\n",
    "print(lda_gensim_para_coherence_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mIiHZ97wddxJ"
   },
   "outputs": [],
   "source": [
    "print(lda_gensim_para_coherence_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BHNZfuVOddxK"
   },
   "outputs": [],
   "source": [
    "lda_gensim_para.alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ltNgzHxLddxK"
   },
   "outputs": [],
   "source": [
    "nmf_gensim_para_coherence = CoherenceModel(model=nmf_gensim_para, texts=gensim_paragraphs, dictionary=dict_gensim_para, coherence='c_v')\n",
    "nmf_gensim_para_coherence_score = nmf_gensim_para_coherence.get_coherence()\n",
    "print(nmf_gensim_para_coherence_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "E_dRCxNxddxK"
   },
   "outputs": [],
   "source": [
    "top_topics = lda_gensim_para.top_topics(vectors_gensim_para, topn=5)\n",
    "avg_topic_coherence = sum([t[1] for t in top_topics]) / len(top_topics)\n",
    "print('Average topic coherence: %.4f.' % avg_topic_coherence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0n-3UFs2ddxK"
   },
   "outputs": [],
   "source": [
    "[(t[1], \" \".join([w[1] for w in t[0]])) for t in top_topics]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mrnFsdWnddxK"
   },
   "source": [
    "## Optimize number of LDA topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LLGVDvLEddxL"
   },
   "outputs": [],
   "source": [
    "from gensim.models.ldamulticore import LdaMulticore\n",
    "lda_para_model_n = []\n",
    "for n in tqdm(range(5, 21)):\n",
    "    lda_model = LdaMulticore(corpus=bow_gensim_para, id2word=dict_gensim_para, chunksize=2000,\n",
    "                             eta='auto', iterations=400, num_topics=n, passes=20, \n",
    "                             eval_every=None, random_state=42)\n",
    "    lda_coherence = CoherenceModel(model=lda_model, texts=gensim_paragraphs, \n",
    "                                   dictionary=dict_gensim_para, coherence='c_v')\n",
    "    lda_para_model_n.append((n, lda_model, lda_coherence.get_coherence()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "XQrp1CHjddxL"
   },
   "outputs": [],
   "source": [
    "pd.DataFrame(lda_para_model_n, columns=[\"n\", \"model\", \"coherence\"]).set_index(\"n\")[[\"coherence\"]].plot(figsize=(16,9))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Zvr8KcafddxL",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "display_topics_gensim(lda_para_model_n[12][1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_nWQG8HcddxM"
   },
   "outputs": [],
   "source": [
    "def wordcloud_topics_gensim(model, no_top_words=40):\n",
    "    for topic in range(0, model.num_topics):\n",
    "        size = {}\n",
    "        for (word, prob) in model.show_topic(topic, topn=no_top_words):\n",
    "            size[word] = prob\n",
    "        wc = WordCloud(background_color=\"white\", max_words=100, width=960, height=540)\n",
    "        wc.generate_from_frequencies(size)\n",
    "        plt.figure(figsize=(12,12))\n",
    "        plt.imshow(wc, interpolation='bilinear')\n",
    "        plt.axis(\"off\")\n",
    "        # if you don't want to save the topic model, comment the next line\n",
    "        plt.savefig(f'topic{topic}.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "aXzMrPQTddxM",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "wordcloud_topics_gensim(lda_para_model_n[12][1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_GCQBFQrddxM"
   },
   "source": [
    "## HDP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "IP7-lx4eddxM"
   },
   "outputs": [],
   "source": [
    "from gensim.models import HdpModel\n",
    "hdp_gensim_para = HdpModel(corpus=bow_gensim_para, id2word=dict_gensim_para)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "T1dJXe7IddxP"
   },
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "83g13zRJddxS"
   },
   "outputs": [],
   "source": [
    "words = 8\n",
    "pd.DataFrame([re.split(r\" \\+ |\\*\", t[1]) for t in hdp_gensim_para.print_topics(num_topics=20, num_words=words)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7xQluJw3ddxS"
   },
   "outputs": [],
   "source": [
    "pd.set_option('max_colwidth', 1000)\n",
    "pd.DataFrame(hdp_gensim_para.print_topics(num_topics=40, num_words=10), columns=[\"topic\", \"words\"]).set_index(\"topic\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3k9vGTCsddxT"
   },
   "outputs": [],
   "source": [
    "hdp_gensim_para.show_topic(0, topn=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "egRDhN_mddxT"
   },
   "outputs": [],
   "source": [
    "def display_topics_gensim_hdp(model, num_topics):\n",
    "    for topic in range(0, num_topics):\n",
    "        print(\"\\nTopic %02d\" % topic)\n",
    "        for (word, prob) in model.show_topic(topic, topn=10):\n",
    "            print(\"  %s (%2.2f)\" % (word, prob))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GB23o18CddxT",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "display_topics_gensim_hdp(hdp_gensim_para, 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lpYRfJkKddxT"
   },
   "source": [
    "# Clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "IPsUc-dpddxU"
   },
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans\n",
    "k_means_para = KMeans(n_clusters=10, random_state=42)\n",
    "k_means_para.fit(tfidf_para_vectors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "F4F-4qOlddxU"
   },
   "outputs": [],
   "source": [
    "sizes = []\n",
    "for i in range(10):\n",
    "    sizes.append({\"cluster\": i, \"size\": np.sum(k_means_para.labels_==i)})\n",
    "pd.DataFrame(sizes).set_index(\"cluster\").plot.bar(figsize=(16,9))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "A_6Q0ZEzddxU"
   },
   "outputs": [],
   "source": [
    "np.unique(k_means_para.labels_, return_counts=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "46jIQfWBddxU"
   },
   "outputs": [],
   "source": [
    "def wordcloud_clusters(model, vectors, features, no_top_words=40):\n",
    "    for cluster in np.unique(model.labels_):\n",
    "        size = {}\n",
    "        words = vectors[model.labels_ == cluster].sum(axis=0).A[0]\n",
    "        largest = words.argsort()[::-1] # invert sort order\n",
    "        for i in range(0, no_top_words):\n",
    "            size[features[largest[i]]] = abs(words[largest[i]])\n",
    "        wc = WordCloud(background_color=\"white\", max_words=100, width=960, height=540)\n",
    "        wc.generate_from_frequencies(size)\n",
    "        plt.figure(figsize=(12,12))\n",
    "        plt.imshow(wc, interpolation='bilinear')\n",
    "        plt.axis(\"off\")\n",
    "        # if you don't want to save the topic model, comment the next line\n",
    "        plt.savefig(f'cluster{cluster}.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "WYcY-h91ddxU",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "wordcloud_clusters(k_means_para, tfidf_para_vectors, tfidf_para_vectorizer.get_feature_names_out())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FPrCmTiWjqXL"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}