lectures/mehrdad/AppendixC.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9befbf07-6eb9-492a-b6a9-eddef01b2eff",
   "metadata": {},
   "source": [
    "# Appendix C: Representing documents using embeddings "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4889a88-0161-47c8-8f11-e2c6dbfe3b7b",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "af690120-6f2f-41dc-8632-1e3d1478ed0c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intel MKL WARNING: Support of Intel(R) Streaming SIMD Extensions 4.2 (Intel(R) SSE4.2) enabled only processors has been deprecated. Intel oneAPI Math Kernel Library 2025.0 will require Intel(R) Advanced Vector Extensions (Intel(R) AVX) instructions.\n",
      "Intel MKL WARNING: Support of Intel(R) Streaming SIMD Extensions 4.2 (Intel(R) SSE4.2) enabled only processors has been deprecated. Intel oneAPI Math Kernel Library 2025.0 will require Intel(R) Advanced Vector Extensions (Intel(R) AVX) instructions.\n"
     ]
    }
   ],
   "source": [
    "## Imports\n",
    "\n",
    "import os\n",
    "import re\n",
    "import string\n",
    "import sys\n",
    "import time\n",
    "\n",
    "sys.path.append(os.path.join(os.path.abspath(\".\"), \"code\"))\n",
    "\n",
    "import IPython\n",
    "import numpy as np\n",
    "import numpy.random as npr\n",
    "import pandas as pd\n",
    "from comat import CooccurrenceMatrix\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.tokenize import sent_tokenize, word_tokenize\n",
    "from preprocessing import MyPreprocessor\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.pipeline import make_pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca0a99de-c938-4f78-b844-dcb09464a571",
   "metadata": {},
   "source": [
    "## (Optional) Representing documents using word embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9773a0f-a6c1-48bf-89af-a4484bb0da57",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Assuming that we have reasonable representations of words. \n",
    "- How do we represent meaning of paragraphs or documents?\n",
    "- Two simple approaches\n",
    "    - Averaging embeddings\n",
    "    - Concatenating embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4561b2e-ce61-4e52-ba97-15325422702d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Averaging embeddings\n",
    "\n",
    "<blockquote>\n",
    "All empty promises\n",
    "</blockquote>\n",
    "    \n",
    "$(embedding(all) + embedding(empty) + embedding(promise))/3$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2eff1762-2dd6-4260-a4e8-475450e9a1c2",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Average embeddings with spaCy\n",
    "\n",
    "- We can do this conveniently with [spaCy](https://spacy.io/usage/linguistic-features#vectors-similarity). \n",
    "- We need `en_core_web_md` model to access word vectors. \n",
    "- You can download the model by going to command line and in your course `conda` environment and download `en_core_web_md` as follows.   \n",
    "\n",
    "```\n",
    "conda activate cpsc330\n",
    "python -m spacy download en_core_web_md\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49ebed77-1f79-422a-9c76-e182e1b3b246",
   "metadata": {},
   "source": [
    "We can access word vectors for individual words in `spaCy` as follows. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b89ab981-0136-42c2-8953-ebc3341a0384",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 6.5486e-01, -2.2584e+00,  6.2793e-02,  1.8801e+00,  2.0700e-01,\n",
       "       -3.3299e+00, -9.6833e-01,  1.5131e+00, -3.7041e+00, -7.7749e-02,\n",
       "        1.5029e+00, -1.7764e+00,  1.7324e+00,  1.6241e+00,  2.6455e-01,\n",
       "       -3.0840e+00,  7.5715e-01, -1.2903e+00,  2.3571e+00, -3.8793e+00,\n",
       "        7.7635e-01,  3.9372e+00,  3.9900e-01, -6.8284e-01, -1.4018e+00,\n",
       "       -2.1673e+00, -1.9244e+00,  1.0629e+00,  3.3378e-01, -8.3864e-01,\n",
       "       -2.5646e-01, -1.7198e+00, -5.4607e-02, -1.4614e+00,  1.3352e+00,\n",
       "       -1.8177e+00,  1.7254e+00,  4.9624e-01,  1.1314e+00, -1.5295e+00,\n",
       "       -8.8629e-01, -2.7562e-01,  7.1799e-01,  1.5554e-01,  3.4230e+00,\n",
       "        2.7167e+00,  1.1793e+00,  2.0961e-01,  3.3121e-01,  1.2322e+00,\n",
       "        1.4375e+00, -4.2099e-01,  6.2814e-01, -1.9051e+00,  3.0593e-02,\n",
       "        6.1895e-01, -3.1495e-01, -2.0444e-04,  2.2073e+00,  3.8856e-01,\n",
       "        1.6554e+00,  1.1932e+00,  2.6678e+00, -5.5454e-01, -1.2078e+00,\n",
       "        1.5709e-01, -1.1324e+00, -2.0163e+00,  1.4567e+00, -2.4244e-01,\n",
       "       -1.9425e+00,  8.3090e-01,  1.7428e-01,  9.1676e-01,  8.8830e-03,\n",
       "        2.4857e-01, -1.2018e+00, -2.3073e+00,  2.2553e+00, -1.5853e+00,\n",
       "       -5.8452e-01,  9.2523e-01, -2.7129e-01, -7.6348e-01,  1.3506e+00,\n",
       "        1.7429e+00,  3.0469e+00,  1.9319e+00, -2.6099e+00,  1.8484e+00,\n",
       "        1.3795e+00,  2.0948e+00,  1.1545e+00, -2.9681e+00, -5.0455e-02,\n",
       "       -5.3864e-01,  2.4820e+00, -1.1131e+00, -2.1827e-01, -2.7559e+00,\n",
       "       -4.4502e-01, -2.8897e+00,  1.7430e+00, -1.5742e+00,  5.7160e-02,\n",
       "        2.4764e+00, -2.5828e+00,  9.3866e-01, -1.3150e+00,  2.3863e+00,\n",
       "        6.1536e-01,  1.7656e-01,  2.0245e+00,  1.6807e-01, -1.2850e+00,\n",
       "        1.6425e-01,  1.7782e+00, -3.2221e+00,  6.1392e-01,  1.3269e+00,\n",
       "       -3.1582e-02,  6.6331e-01, -6.8109e-01,  5.0985e-01, -4.2942e-01,\n",
       "       -1.6438e-01,  7.9306e-01, -3.0776e+00,  1.8022e+00, -4.5356e-01,\n",
       "       -1.6405e+00,  8.1761e-01,  1.4960e+00, -6.2266e-01,  8.5264e-01,\n",
       "       -5.0226e-01, -1.5735e+00, -4.5090e+00, -5.0587e-01, -1.5471e+00,\n",
       "       -5.3910e-01, -6.6574e-01,  7.6376e-01, -1.4926e+00, -7.8819e-01,\n",
       "       -9.9256e-01,  1.1512e+00,  5.2091e-01,  1.6460e-01, -2.6747e+00,\n",
       "       -1.7082e+00,  1.5789e+00, -2.8982e-01, -1.2842e+00, -1.1286e+00,\n",
       "        7.6392e-01,  3.2199e+00,  7.5850e-01,  1.3628e+00, -1.3231e+00,\n",
       "        2.2350e-02, -2.5602e+00,  6.7751e-01,  4.0511e-01,  1.8997e+00,\n",
       "       -1.1051e+00, -1.3014e-01,  7.2024e-01,  6.2354e-02,  1.1913e-01,\n",
       "       -1.1978e+00, -1.5625e+00, -2.5975e-01,  2.5911e+00, -3.2413e+00,\n",
       "       -3.8988e-01, -4.0542e-01, -1.8894e+00,  3.4278e+00, -3.3625e-01,\n",
       "       -2.0979e+00,  1.3275e+00, -2.0514e+00,  2.4583e-01, -7.3326e-01,\n",
       "       -2.3684e+00,  2.8493e+00, -5.2075e-01,  2.2708e-01, -6.8701e-01,\n",
       "       -7.0855e-01, -7.5334e-01,  7.3050e-02,  2.2246e+00, -2.6824e-01,\n",
       "       -2.8289e-01, -1.8230e+00,  2.2047e+00, -2.4848e-01, -2.3042e-02,\n",
       "        1.0358e+00, -2.7074e-01, -5.6816e-02, -9.1017e-01,  1.2943e-01,\n",
       "       -1.4274e+00, -3.6128e-01,  7.3127e-01,  2.0264e+00,  7.2928e-01,\n",
       "        1.7298e+00,  1.1075e+00, -7.0250e-01,  1.6928e+00,  2.0074e+00,\n",
       "       -7.5464e-01,  1.6378e+00,  3.5970e-01, -2.2128e-01, -1.7607e-01,\n",
       "        1.8260e+00, -2.5962e-01, -1.4320e+00,  7.8332e-01,  2.1438e+00,\n",
       "       -2.4723e+00, -1.4913e-01,  6.2585e-01,  6.6819e-01,  2.3947e+00,\n",
       "       -2.7173e+00,  2.4134e-03, -8.6530e-01, -9.7728e-01, -2.9815e+00,\n",
       "        1.6895e+00, -7.1146e-01,  3.2025e+00, -9.4129e-01, -1.9695e+00,\n",
       "        7.7711e-01, -3.2278e-01, -1.3727e+00,  2.9276e+00, -1.5440e-01,\n",
       "        1.7169e+00,  5.5736e-01,  1.4620e-01, -1.1244e+00, -2.4633e+00,\n",
       "       -2.2685e+00,  1.2459e+00, -2.0362e+00, -4.8331e-01, -6.3194e-01,\n",
       "       -2.4082e+00, -9.0132e-01,  3.0541e+00, -2.2632e+00, -3.7800e-01,\n",
       "       -3.1647e-01,  1.0785e+00, -3.0444e-01,  1.2112e+00, -1.3496e+00,\n",
       "        1.0599e+00,  4.2607e-01,  4.0194e-01, -2.8586e+00,  1.0107e+00,\n",
       "        1.5924e+00, -5.1770e-01,  1.3246e+00,  3.2268e-01, -1.3978e-01,\n",
       "       -2.1841e+00,  1.6548e+00,  1.3903e+00,  6.3376e-01, -4.7083e-01,\n",
       "        6.8377e-01, -1.3031e+00, -1.3292e-01, -1.1567e+00,  5.3419e-01,\n",
       "       -1.3412e+00, -1.5887e+00, -9.4468e-01, -2.4031e+00,  3.1785e+00,\n",
       "        1.1524e+00, -1.1699e+00,  9.8752e-01, -1.0660e+00, -2.1852e+00,\n",
       "       -3.1228e-01,  3.0012e+00, -1.2234e+00,  5.7454e-01, -2.1885e-01],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_md\")\n",
    "\n",
    "doc = nlp(\"pineapple\") # extract all interesting information about the document\n",
    "doc.vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "109143ab-2a6f-4049-b602-f3e595ac31ec",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.010289,  4.9203  , -0.48081 ,  3.5738  , -2.2516  ,  2.1697  ,\n",
       "       -1.0116  ,  2.4216  , -3.7343  ,  3.3025  ], dtype=float32)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp(\"empty\").vector[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5f580bb-589b-4c36-a578-7b1d5a7db033",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can get average embeddings for a sentence or a document in `spaCy` as follows: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7ebc8bbc-fc20-46a2-97c1-7d05d316eeac",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(300,)\n",
      "Vector for: All empty promises\n",
      "[-0.459937    1.9785299   1.0319      1.5123      1.4806334   2.73183\n",
      "  1.204       1.1724668  -3.5227966  -0.05656664]\n"
     ]
    }
   ],
   "source": [
    "s = \"All empty promises\"\n",
    "doc = nlp(s)\n",
    "avg_sent_emb = doc.vector\n",
    "print(avg_sent_emb.shape)\n",
    "print(\"Vector for: {}\\n{}\".format((s), (avg_sent_emb[0:10])))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d9c9bdc-889c-414f-af2d-bc6f46c7722d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Similarity between documents \n",
    "\n",
    "- We can also get similarity between documents as follows. \n",
    "- Note that this is based on average embeddings of each sentence. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d6617a9b-fc0c-4fe5-98d1-f4f672715cf9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deep learning is very popular these days. <-> Machine learning is dominated by neural networks. 0.699868820717508\n",
      "Machine learning is dominated by neural networks. <-> A home-made fresh bread with butter and cheese. 0.5098293421139041\n"
     ]
    }
   ],
   "source": [
    "doc1 = nlp(\"Deep learning is very popular these days.\")\n",
    "doc2 = nlp(\"Machine learning is dominated by neural networks.\")\n",
    "doc3 = nlp(\"A home-made fresh bread with butter and cheese.\")\n",
    "\n",
    "# Similarity of two documents\n",
    "print(doc1, \"<->\", doc2, doc1.similarity(doc2))\n",
    "print(doc2, \"<->\", doc3, doc2.similarity(doc3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9abccc0a-3e83-40d4-b6a5-06e8e37fb0e6",
   "metadata": {},
   "source": [
    "- Do these scores make sense? \n",
    "- There are no common words, but we are still able to identify that doc1 and doc2 are more similar that doc2 and doc3.  \n",
    "- You can use such average embedding representation in text classification tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcdfa6d9-6f4f-4ac9-b4c2-8dac7730d2fb",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "###  Airline sentiment analysis using average embedding representation \n",
    "\n",
    "- Let's try average embedding representation for airline sentiment analysis.\n",
    "- You can download the data [here](https://www.kaggle.com/jaskarancr/airline-sentiment-dataset). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "246cbc6e-f902-496b-b4cf-42fecc387ca0",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"data/Airline-Sentiment-2-w-AA.csv\", encoding=\"ISO-8859-1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8ad0259e-9079-4a20-9990-fb37df57ce2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_validate, train_test_split\n",
    "\n",
    "train_df, test_df = train_test_split(df, test_size=0.2, random_state=123)\n",
    "X_train, y_train = train_df[\"text\"], train_df[\"airline_sentiment\"]\n",
    "X_test, y_test = test_df[\"text\"], test_df[\"airline_sentiment\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "40b2c92c-de01-4d51-be0e-2e80f3f08f65",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_unit_id</th>\n",
       "      <th>_golden</th>\n",
       "      <th>_unit_state</th>\n",
       "      <th>_trusted_judgments</th>\n",
       "      <th>_last_judgment_at</th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>airline_sentiment:confidence</th>\n",
       "      <th>negativereason</th>\n",
       "      <th>negativereason:confidence</th>\n",
       "      <th>airline</th>\n",
       "      <th>airline_sentiment_gold</th>\n",
       "      <th>name</th>\n",
       "      <th>negativereason_gold</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>text</th>\n",
       "      <th>tweet_coord</th>\n",
       "      <th>tweet_created</th>\n",
       "      <th>tweet_id</th>\n",
       "      <th>tweet_location</th>\n",
       "      <th>user_timezone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5789</th>\n",
       "      <td>681455792</td>\n",
       "      <td>False</td>\n",
       "      <td>finalized</td>\n",
       "      <td>3</td>\n",
       "      <td>2/25/15 4:21</td>\n",
       "      <td>negative</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Can't Tell</td>\n",
       "      <td>0.6667</td>\n",
       "      <td>Southwest</td>\n",
       "      <td>NaN</td>\n",
       "      <td>mrssuperdimmock</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@SouthwestAir link doesn't work</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2/19/15 18:53</td>\n",
       "      <td>5.686040e+17</td>\n",
       "      <td>Lake Arrowhead, CA</td>\n",
       "      <td>Pacific Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8918</th>\n",
       "      <td>681459957</td>\n",
       "      <td>False</td>\n",
       "      <td>finalized</td>\n",
       "      <td>3</td>\n",
       "      <td>2/25/15 9:45</td>\n",
       "      <td>neutral</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Delta</td>\n",
       "      <td>NaN</td>\n",
       "      <td>labeles</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@JetBlue okayyyy. But I had huge irons on way ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2/17/15 10:18</td>\n",
       "      <td>5.677500e+17</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11688</th>\n",
       "      <td>681462990</td>\n",
       "      <td>False</td>\n",
       "      <td>finalized</td>\n",
       "      <td>3</td>\n",
       "      <td>2/25/15 9:53</td>\n",
       "      <td>negative</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Customer Service Issue</td>\n",
       "      <td>0.6727</td>\n",
       "      <td>US Airways</td>\n",
       "      <td>NaN</td>\n",
       "      <td>DropMeAnywhere</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@USAirways They're all reservations numbers an...</td>\n",
       "      <td>[0.0, 0.0]</td>\n",
       "      <td>2/17/15 14:50</td>\n",
       "      <td>5.678190e+17</td>\n",
       "      <td>Here, There and Everywhere</td>\n",
       "      <td>Arizona</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>413</th>\n",
       "      <td>681448905</td>\n",
       "      <td>False</td>\n",
       "      <td>finalized</td>\n",
       "      <td>3</td>\n",
       "      <td>2/25/15 10:10</td>\n",
       "      <td>neutral</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>jsamaudio</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica no A's channel this year?</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2/18/15 12:25</td>\n",
       "      <td>5.681440e+17</td>\n",
       "      <td>St. Francis (Calif.)</td>\n",
       "      <td>Pacific Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4135</th>\n",
       "      <td>681454122</td>\n",
       "      <td>False</td>\n",
       "      <td>finalized</td>\n",
       "      <td>3</td>\n",
       "      <td>2/25/15 10:08</td>\n",
       "      <td>negative</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Bad Flight</td>\n",
       "      <td>0.3544</td>\n",
       "      <td>United</td>\n",
       "      <td>NaN</td>\n",
       "      <td>CajunSQL</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@united missed it.  Incoming on time, then Sat...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2/17/15 14:20</td>\n",
       "      <td>5.678110e+17</td>\n",
       "      <td>Baton Rouge, LA</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        _unit_id  _golden _unit_state  _trusted_judgments _last_judgment_at  \\\n",
       "5789   681455792    False   finalized                   3      2/25/15 4:21   \n",
       "8918   681459957    False   finalized                   3      2/25/15 9:45   \n",
       "11688  681462990    False   finalized                   3      2/25/15 9:53   \n",
       "413    681448905    False   finalized                   3     2/25/15 10:10   \n",
       "4135   681454122    False   finalized                   3     2/25/15 10:08   \n",
       "\n",
       "      airline_sentiment  airline_sentiment:confidence          negativereason  \\\n",
       "5789           negative                           1.0              Can't Tell   \n",
       "8918            neutral                           1.0                     NaN   \n",
       "11688          negative                           1.0  Customer Service Issue   \n",
       "413             neutral                           1.0                     NaN   \n",
       "4135           negative                           1.0              Bad Flight   \n",
       "\n",
       "       negativereason:confidence         airline airline_sentiment_gold  \\\n",
       "5789                      0.6667       Southwest                    NaN   \n",
       "8918                         NaN           Delta                    NaN   \n",
       "11688                     0.6727      US Airways                    NaN   \n",
       "413                          NaN  Virgin America                    NaN   \n",
       "4135                      0.3544          United                    NaN   \n",
       "\n",
       "                  name negativereason_gold  retweet_count  \\\n",
       "5789   mrssuperdimmock                 NaN              0   \n",
       "8918           labeles                 NaN              0   \n",
       "11688   DropMeAnywhere                 NaN              0   \n",
       "413          jsamaudio                 NaN              0   \n",
       "4135          CajunSQL                 NaN              0   \n",
       "\n",
       "                                                    text tweet_coord  \\\n",
       "5789                     @SouthwestAir link doesn't work         NaN   \n",
       "8918   @JetBlue okayyyy. But I had huge irons on way ...         NaN   \n",
       "11688  @USAirways They're all reservations numbers an...  [0.0, 0.0]   \n",
       "413             @VirginAmerica no A's channel this year?         NaN   \n",
       "4135   @united missed it.  Incoming on time, then Sat...         NaN   \n",
       "\n",
       "       tweet_created      tweet_id              tweet_location  \\\n",
       "5789   2/19/15 18:53  5.686040e+17          Lake Arrowhead, CA   \n",
       "8918   2/17/15 10:18  5.677500e+17                         NaN   \n",
       "11688  2/17/15 14:50  5.678190e+17  Here, There and Everywhere   \n",
       "413    2/18/15 12:25  5.681440e+17        St. Francis (Calif.)   \n",
       "4135   2/17/15 14:20  5.678110e+17             Baton Rouge, LA   \n",
       "\n",
       "                    user_timezone  \n",
       "5789   Pacific Time (US & Canada)  \n",
       "8918                          NaN  \n",
       "11688                     Arizona  \n",
       "413    Pacific Time (US & Canada)  \n",
       "4135                          NaN  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65eb32ba-839b-4343-8136-acd7c7beb50a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Bag-of-words representation for sentiment analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2591739d-2b7a-4be4-a95a-2e82567e0432",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data matrix shape: (11712, 13064)\n"
     ]
    }
   ],
   "source": [
    "pipe = make_pipeline(\n",
    "    CountVectorizer(stop_words=\"english\"), LogisticRegression(max_iter=1000)\n",
    ")\n",
    "pipe.named_steps[\"countvectorizer\"].fit(X_train)\n",
    "X_train_transformed = pipe.named_steps[\"countvectorizer\"].transform(X_train)\n",
    "print(\"Data matrix shape:\", X_train_transformed.shape)\n",
    "pipe.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0864e048-dd40-42ca-899f-a0a9d607c142",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy 0.94\n",
      "Test accuracy 0.80\n"
     ]
    }
   ],
   "source": [
    "print(\"Train accuracy {:.2f}\".format(pipe.score(X_train, y_train)))\n",
    "print(\"Test accuracy {:.2f}\".format(pipe.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f47bf17d-c456-4ca0-bc28-9b114cb3f1c4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Sentiment analysis with average embedding representation\n",
    "\n",
    "- Let's see how can we get word vectors using `spaCy`. \n",
    "- Let's create average embedding representation for each example. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "44ba2026-41c2-4d4b-8bc1-bdd4b77fe808",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "X_train_embeddings = pd.DataFrame([text.vector for text in nlp.pipe(X_train)])\n",
    "X_test_embeddings = pd.DataFrame([text.vector for text in nlp.pipe(X_test)])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "582d7ec3-1e53-40d6-a703-0c642416ea69",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "We have reduced dimensionality from 13,064 to 300! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "dbef265e-f6e2-46ff-afe2-d18cf572de4f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(11712, 300)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_embeddings.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e4fa5e91-5d2c-4d3d-999d-8571955242a4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>290</th>\n",
       "      <th>291</th>\n",
       "      <th>292</th>\n",
       "      <th>293</th>\n",
       "      <th>294</th>\n",
       "      <th>295</th>\n",
       "      <th>296</th>\n",
       "      <th>297</th>\n",
       "      <th>298</th>\n",
       "      <th>299</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.259942</td>\n",
       "      <td>4.856640</td>\n",
       "      <td>-2.677500</td>\n",
       "      <td>-1.875390</td>\n",
       "      <td>0.459240</td>\n",
       "      <td>-0.304300</td>\n",
       "      <td>3.273020</td>\n",
       "      <td>2.008458</td>\n",
       "      <td>-4.818360</td>\n",
       "      <td>1.002320</td>\n",
       "      <td>...</td>\n",
       "      <td>1.555840</td>\n",
       "      <td>1.274170</td>\n",
       "      <td>0.716420</td>\n",
       "      <td>-1.296220</td>\n",
       "      <td>-1.403746</td>\n",
       "      <td>0.430054</td>\n",
       "      <td>1.107044</td>\n",
       "      <td>0.224080</td>\n",
       "      <td>-0.903256</td>\n",
       "      <td>-0.295254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.563826</td>\n",
       "      <td>0.869816</td>\n",
       "      <td>-2.462877</td>\n",
       "      <td>0.057751</td>\n",
       "      <td>0.892089</td>\n",
       "      <td>0.566698</td>\n",
       "      <td>-0.283121</td>\n",
       "      <td>3.594112</td>\n",
       "      <td>-1.578107</td>\n",
       "      <td>1.037976</td>\n",
       "      <td>...</td>\n",
       "      <td>0.270313</td>\n",
       "      <td>-0.948990</td>\n",
       "      <td>2.653996</td>\n",
       "      <td>-1.631425</td>\n",
       "      <td>-1.850329</td>\n",
       "      <td>0.432467</td>\n",
       "      <td>0.979464</td>\n",
       "      <td>1.015146</td>\n",
       "      <td>-2.604236</td>\n",
       "      <td>-0.188137</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.707503</td>\n",
       "      <td>0.908782</td>\n",
       "      <td>-3.248327</td>\n",
       "      <td>-0.667797</td>\n",
       "      <td>2.590958</td>\n",
       "      <td>2.246804</td>\n",
       "      <td>-1.095754</td>\n",
       "      <td>3.953429</td>\n",
       "      <td>-1.524224</td>\n",
       "      <td>0.593270</td>\n",
       "      <td>...</td>\n",
       "      <td>0.948402</td>\n",
       "      <td>-1.760466</td>\n",
       "      <td>1.369070</td>\n",
       "      <td>-2.204359</td>\n",
       "      <td>-1.357214</td>\n",
       "      <td>0.389199</td>\n",
       "      <td>0.205336</td>\n",
       "      <td>0.100318</td>\n",
       "      <td>-2.950879</td>\n",
       "      <td>1.399726</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.977736</td>\n",
       "      <td>4.676425</td>\n",
       "      <td>-0.224362</td>\n",
       "      <td>-1.011286</td>\n",
       "      <td>3.981441</td>\n",
       "      <td>-1.132660</td>\n",
       "      <td>0.988456</td>\n",
       "      <td>2.997113</td>\n",
       "      <td>0.553975</td>\n",
       "      <td>-0.539687</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.229152</td>\n",
       "      <td>1.099266</td>\n",
       "      <td>1.652113</td>\n",
       "      <td>0.036738</td>\n",
       "      <td>0.315839</td>\n",
       "      <td>1.429018</td>\n",
       "      <td>0.557437</td>\n",
       "      <td>-0.333650</td>\n",
       "      <td>-0.281916</td>\n",
       "      <td>-0.587222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.725984</td>\n",
       "      <td>0.178514</td>\n",
       "      <td>-1.163662</td>\n",
       "      <td>0.597000</td>\n",
       "      <td>4.621603</td>\n",
       "      <td>-1.166608</td>\n",
       "      <td>1.256955</td>\n",
       "      <td>3.940082</td>\n",
       "      <td>0.530063</td>\n",
       "      <td>-0.816050</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.398607</td>\n",
       "      <td>-0.219403</td>\n",
       "      <td>0.925384</td>\n",
       "      <td>-0.751906</td>\n",
       "      <td>-1.969212</td>\n",
       "      <td>1.381326</td>\n",
       "      <td>2.146303</td>\n",
       "      <td>0.195811</td>\n",
       "      <td>-2.914482</td>\n",
       "      <td>0.297692</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 300 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        0         1         2         3         4         5         6    \\\n",
       "0  1.259942  4.856640 -2.677500 -1.875390  0.459240 -0.304300  3.273020   \n",
       "1 -0.563826  0.869816 -2.462877  0.057751  0.892089  0.566698 -0.283121   \n",
       "2 -0.707503  0.908782 -3.248327 -0.667797  2.590958  2.246804 -1.095754   \n",
       "3 -0.977736  4.676425 -0.224362 -1.011286  3.981441 -1.132660  0.988456   \n",
       "4 -0.725984  0.178514 -1.163662  0.597000  4.621603 -1.166608  1.256955   \n",
       "\n",
       "        7         8         9    ...       290       291       292       293  \\\n",
       "0  2.008458 -4.818360  1.002320  ...  1.555840  1.274170  0.716420 -1.296220   \n",
       "1  3.594112 -1.578107  1.037976  ...  0.270313 -0.948990  2.653996 -1.631425   \n",
       "2  3.953429 -1.524224  0.593270  ...  0.948402 -1.760466  1.369070 -2.204359   \n",
       "3  2.997113  0.553975 -0.539687  ... -0.229152  1.099266  1.652113  0.036738   \n",
       "4  3.940082  0.530063 -0.816050  ... -0.398607 -0.219403  0.925384 -0.751906   \n",
       "\n",
       "        294       295       296       297       298       299  \n",
       "0 -1.403746  0.430054  1.107044  0.224080 -0.903256 -0.295254  \n",
       "1 -1.850329  0.432467  0.979464  1.015146 -2.604236 -0.188137  \n",
       "2 -1.357214  0.389199  0.205336  0.100318 -2.950879  1.399726  \n",
       "3  0.315839  1.429018  0.557437 -0.333650 -0.281916 -0.587222  \n",
       "4 -1.969212  1.381326  2.146303  0.195811 -2.914482  0.297692  \n",
       "\n",
       "[5 rows x 300 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_embeddings.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "cdf7fe24-3e64-40ac-aa2d-7fabbffd4a2f",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy 0.80\n",
      "Test accuracy 0.79\n"
     ]
    }
   ],
   "source": [
    "lgr = LogisticRegression(max_iter=2000)\n",
    "lgr.fit(X_train_embeddings, y_train)\n",
    "print(\"Train accuracy {:.2f}\".format(lgr.score(X_train_embeddings, y_train)))\n",
    "print(\"Test accuracy {:.2f}\".format(lgr.score(X_test_embeddings, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13df5f93-64fa-4b8f-ad14-8c8d9784379c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Sentiment classification using average embeddings \n",
    "\n",
    "- What are the train and test accuracies with average word embedding representation? \n",
    "- The accuracy is similar with less overfitting. \n",
    "- Note that we are using **transfer learning** here.\n",
    "- The embeddings are trained on a completely different corpus. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75a3067c-fd2d-40af-a6a0-2429debb448d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### (Optional) Sentiment classification using advanced sentence representations "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07639298-df35-45f2-bca7-407d29626a14",
   "metadata": {},
   "source": [
    "- Since, representing documents is so essential for text classification tasks, there are more advanced methods for document representation. \n",
    "- In homework 6, you also explore sentence embedding representation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "b569e81e-f694-4831-8864-f38b61e7604f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sentence_transformers import SentenceTransformer\n",
    "\n",
    "embedder = SentenceTransformer(\"paraphrase-distilroberta-base-v1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "81be1284-1a5a-4766-87c9-f249e8ecfd8c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(768,)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emb_sents = embedder.encode(\"all empty promises\")\n",
    "emb_sents.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "63c6bdbb-99d2-41c7-b90f-eae9d76e8b46",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>758</th>\n",
       "      <th>759</th>\n",
       "      <th>760</th>\n",
       "      <th>761</th>\n",
       "      <th>762</th>\n",
       "      <th>763</th>\n",
       "      <th>764</th>\n",
       "      <th>765</th>\n",
       "      <th>766</th>\n",
       "      <th>767</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5789</th>\n",
       "      <td>-0.120494</td>\n",
       "      <td>0.250263</td>\n",
       "      <td>-0.022795</td>\n",
       "      <td>-0.116368</td>\n",
       "      <td>0.078650</td>\n",
       "      <td>0.037357</td>\n",
       "      <td>-0.251341</td>\n",
       "      <td>0.321429</td>\n",
       "      <td>-0.143984</td>\n",
       "      <td>-0.123486</td>\n",
       "      <td>...</td>\n",
       "      <td>0.199150</td>\n",
       "      <td>-0.150143</td>\n",
       "      <td>0.167078</td>\n",
       "      <td>-0.407671</td>\n",
       "      <td>-0.066161</td>\n",
       "      <td>0.049514</td>\n",
       "      <td>0.019384</td>\n",
       "      <td>-0.357602</td>\n",
       "      <td>0.125996</td>\n",
       "      <td>0.381074</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8918</th>\n",
       "      <td>-0.182954</td>\n",
       "      <td>0.118282</td>\n",
       "      <td>0.066341</td>\n",
       "      <td>-0.136098</td>\n",
       "      <td>0.094947</td>\n",
       "      <td>-0.121303</td>\n",
       "      <td>0.069233</td>\n",
       "      <td>-0.097500</td>\n",
       "      <td>0.025740</td>\n",
       "      <td>-0.367981</td>\n",
       "      <td>...</td>\n",
       "      <td>0.113612</td>\n",
       "      <td>0.114662</td>\n",
       "      <td>0.049926</td>\n",
       "      <td>0.256736</td>\n",
       "      <td>-0.118687</td>\n",
       "      <td>-0.190720</td>\n",
       "      <td>0.011985</td>\n",
       "      <td>-0.141883</td>\n",
       "      <td>-0.230142</td>\n",
       "      <td>0.024899</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11688</th>\n",
       "      <td>-0.032988</td>\n",
       "      <td>0.630251</td>\n",
       "      <td>-0.079516</td>\n",
       "      <td>0.148981</td>\n",
       "      <td>0.194708</td>\n",
       "      <td>-0.226263</td>\n",
       "      <td>-0.043630</td>\n",
       "      <td>0.217398</td>\n",
       "      <td>-0.010715</td>\n",
       "      <td>0.069644</td>\n",
       "      <td>...</td>\n",
       "      <td>0.676791</td>\n",
       "      <td>0.244484</td>\n",
       "      <td>0.051042</td>\n",
       "      <td>0.064099</td>\n",
       "      <td>-0.146945</td>\n",
       "      <td>0.090878</td>\n",
       "      <td>-0.090060</td>\n",
       "      <td>0.077211</td>\n",
       "      <td>-0.209226</td>\n",
       "      <td>0.308773</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>413</th>\n",
       "      <td>-0.119258</td>\n",
       "      <td>0.172168</td>\n",
       "      <td>0.098697</td>\n",
       "      <td>0.319858</td>\n",
       "      <td>0.415475</td>\n",
       "      <td>0.248359</td>\n",
       "      <td>-0.025923</td>\n",
       "      <td>0.385350</td>\n",
       "      <td>0.066414</td>\n",
       "      <td>-0.334289</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.128482</td>\n",
       "      <td>-0.232446</td>\n",
       "      <td>-0.077805</td>\n",
       "      <td>0.181329</td>\n",
       "      <td>0.123244</td>\n",
       "      <td>-0.143693</td>\n",
       "      <td>0.660457</td>\n",
       "      <td>-0.048714</td>\n",
       "      <td>0.204774</td>\n",
       "      <td>0.163497</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4135</th>\n",
       "      <td>0.094240</td>\n",
       "      <td>0.360193</td>\n",
       "      <td>0.213747</td>\n",
       "      <td>0.363690</td>\n",
       "      <td>0.275521</td>\n",
       "      <td>0.134936</td>\n",
       "      <td>-0.276319</td>\n",
       "      <td>0.009336</td>\n",
       "      <td>-0.021523</td>\n",
       "      <td>-0.258992</td>\n",
       "      <td>...</td>\n",
       "      <td>0.474885</td>\n",
       "      <td>0.242125</td>\n",
       "      <td>0.294533</td>\n",
       "      <td>0.279013</td>\n",
       "      <td>0.037831</td>\n",
       "      <td>0.089761</td>\n",
       "      <td>-0.548748</td>\n",
       "      <td>-0.049258</td>\n",
       "      <td>0.154525</td>\n",
       "      <td>0.141268</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5218</th>\n",
       "      <td>-0.204408</td>\n",
       "      <td>-0.145289</td>\n",
       "      <td>-0.064201</td>\n",
       "      <td>0.213571</td>\n",
       "      <td>-0.140225</td>\n",
       "      <td>0.338556</td>\n",
       "      <td>-0.148578</td>\n",
       "      <td>0.224515</td>\n",
       "      <td>-0.042963</td>\n",
       "      <td>0.075930</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.161948</td>\n",
       "      <td>0.040582</td>\n",
       "      <td>0.003971</td>\n",
       "      <td>-0.152549</td>\n",
       "      <td>-0.582907</td>\n",
       "      <td>-0.126526</td>\n",
       "      <td>0.060502</td>\n",
       "      <td>-0.111495</td>\n",
       "      <td>-0.097493</td>\n",
       "      <td>0.199321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12252</th>\n",
       "      <td>0.108408</td>\n",
       "      <td>0.438293</td>\n",
       "      <td>0.216812</td>\n",
       "      <td>-0.349289</td>\n",
       "      <td>0.422689</td>\n",
       "      <td>0.377760</td>\n",
       "      <td>0.045198</td>\n",
       "      <td>-0.034095</td>\n",
       "      <td>0.427570</td>\n",
       "      <td>-0.328272</td>\n",
       "      <td>...</td>\n",
       "      <td>0.257849</td>\n",
       "      <td>-0.032363</td>\n",
       "      <td>-0.275004</td>\n",
       "      <td>0.080452</td>\n",
       "      <td>-0.078975</td>\n",
       "      <td>-0.049972</td>\n",
       "      <td>-0.009761</td>\n",
       "      <td>-0.314754</td>\n",
       "      <td>-0.020773</td>\n",
       "      <td>0.268777</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1346</th>\n",
       "      <td>0.068411</td>\n",
       "      <td>0.017591</td>\n",
       "      <td>0.236154</td>\n",
       "      <td>0.221446</td>\n",
       "      <td>-0.103568</td>\n",
       "      <td>0.055510</td>\n",
       "      <td>0.062910</td>\n",
       "      <td>0.067424</td>\n",
       "      <td>-0.003504</td>\n",
       "      <td>-0.157757</td>\n",
       "      <td>...</td>\n",
       "      <td>0.007711</td>\n",
       "      <td>0.323297</td>\n",
       "      <td>0.334637</td>\n",
       "      <td>0.367041</td>\n",
       "      <td>-0.068821</td>\n",
       "      <td>0.063667</td>\n",
       "      <td>-0.329990</td>\n",
       "      <td>0.232330</td>\n",
       "      <td>-0.184768</td>\n",
       "      <td>-0.000682</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11646</th>\n",
       "      <td>-0.091488</td>\n",
       "      <td>-0.155708</td>\n",
       "      <td>0.032391</td>\n",
       "      <td>0.018314</td>\n",
       "      <td>0.524997</td>\n",
       "      <td>0.563933</td>\n",
       "      <td>-0.080985</td>\n",
       "      <td>0.097982</td>\n",
       "      <td>-0.535285</td>\n",
       "      <td>-0.377194</td>\n",
       "      <td>...</td>\n",
       "      <td>0.428013</td>\n",
       "      <td>-0.144572</td>\n",
       "      <td>0.045297</td>\n",
       "      <td>-0.107935</td>\n",
       "      <td>-0.135673</td>\n",
       "      <td>-0.290019</td>\n",
       "      <td>-0.137200</td>\n",
       "      <td>-0.503395</td>\n",
       "      <td>-0.042567</td>\n",
       "      <td>-0.282592</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3582</th>\n",
       "      <td>0.185626</td>\n",
       "      <td>0.092904</td>\n",
       "      <td>0.097085</td>\n",
       "      <td>-0.174650</td>\n",
       "      <td>-0.193584</td>\n",
       "      <td>0.047294</td>\n",
       "      <td>0.098216</td>\n",
       "      <td>0.332670</td>\n",
       "      <td>0.163098</td>\n",
       "      <td>-0.135101</td>\n",
       "      <td>...</td>\n",
       "      <td>0.078530</td>\n",
       "      <td>-0.030177</td>\n",
       "      <td>0.391598</td>\n",
       "      <td>0.073520</td>\n",
       "      <td>-0.454037</td>\n",
       "      <td>-0.244358</td>\n",
       "      <td>-0.790682</td>\n",
       "      <td>-0.607010</td>\n",
       "      <td>-0.255162</td>\n",
       "      <td>0.029779</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>11712 rows × 768 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            0         1         2         3         4         5         6    \\\n",
       "5789  -0.120494  0.250263 -0.022795 -0.116368  0.078650  0.037357 -0.251341   \n",
       "8918  -0.182954  0.118282  0.066341 -0.136098  0.094947 -0.121303  0.069233   \n",
       "11688 -0.032988  0.630251 -0.079516  0.148981  0.194708 -0.226263 -0.043630   \n",
       "413   -0.119258  0.172168  0.098697  0.319858  0.415475  0.248359 -0.025923   \n",
       "4135   0.094240  0.360193  0.213747  0.363690  0.275521  0.134936 -0.276319   \n",
       "...         ...       ...       ...       ...       ...       ...       ...   \n",
       "5218  -0.204408 -0.145289 -0.064201  0.213571 -0.140225  0.338556 -0.148578   \n",
       "12252  0.108408  0.438293  0.216812 -0.349289  0.422689  0.377760  0.045198   \n",
       "1346   0.068411  0.017591  0.236154  0.221446 -0.103568  0.055510  0.062910   \n",
       "11646 -0.091488 -0.155708  0.032391  0.018314  0.524997  0.563933 -0.080985   \n",
       "3582   0.185626  0.092904  0.097085 -0.174650 -0.193584  0.047294  0.098216   \n",
       "\n",
       "            7         8         9    ...       758       759       760  \\\n",
       "5789   0.321429 -0.143984 -0.123486  ...  0.199150 -0.150143  0.167078   \n",
       "8918  -0.097500  0.025740 -0.367981  ...  0.113612  0.114662  0.049926   \n",
       "11688  0.217398 -0.010715  0.069644  ...  0.676791  0.244484  0.051042   \n",
       "413    0.385350  0.066414 -0.334289  ... -0.128482 -0.232446 -0.077805   \n",
       "4135   0.009336 -0.021523 -0.258992  ...  0.474885  0.242125  0.294533   \n",
       "...         ...       ...       ...  ...       ...       ...       ...   \n",
       "5218   0.224515 -0.042963  0.075930  ... -0.161948  0.040582  0.003971   \n",
       "12252 -0.034095  0.427570 -0.328272  ...  0.257849 -0.032363 -0.275004   \n",
       "1346   0.067424 -0.003504 -0.157757  ...  0.007711  0.323297  0.334637   \n",
       "11646  0.097982 -0.535285 -0.377194  ...  0.428013 -0.144572  0.045297   \n",
       "3582   0.332670  0.163098 -0.135101  ...  0.078530 -0.030177  0.391598   \n",
       "\n",
       "            761       762       763       764       765       766       767  \n",
       "5789  -0.407671 -0.066161  0.049514  0.019384 -0.357602  0.125996  0.381074  \n",
       "8918   0.256736 -0.118687 -0.190720  0.011985 -0.141883 -0.230142  0.024899  \n",
       "11688  0.064099 -0.146945  0.090878 -0.090060  0.077211 -0.209226  0.308773  \n",
       "413    0.181329  0.123244 -0.143693  0.660457 -0.048714  0.204774  0.163497  \n",
       "4135   0.279013  0.037831  0.089761 -0.548748 -0.049258  0.154525  0.141268  \n",
       "...         ...       ...       ...       ...       ...       ...       ...  \n",
       "5218  -0.152549 -0.582907 -0.126526  0.060502 -0.111495 -0.097493  0.199321  \n",
       "12252  0.080452 -0.078975 -0.049972 -0.009761 -0.314754 -0.020773  0.268777  \n",
       "1346   0.367041 -0.068821  0.063667 -0.329990  0.232330 -0.184768 -0.000682  \n",
       "11646 -0.107935 -0.135673 -0.290019 -0.137200 -0.503395 -0.042567 -0.282592  \n",
       "3582   0.073520 -0.454037 -0.244358 -0.790682 -0.607010 -0.255162  0.029779  \n",
       "\n",
       "[11712 rows x 768 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emb_train = embedder.encode(train_df[\"text\"].tolist())\n",
    "emb_train_df = pd.DataFrame(emb_train, index=train_df.index)\n",
    "emb_train_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "e553bd47-97ce-4d55-88b4-c91ce2ae49bc",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>758</th>\n",
       "      <th>759</th>\n",
       "      <th>760</th>\n",
       "      <th>761</th>\n",
       "      <th>762</th>\n",
       "      <th>763</th>\n",
       "      <th>764</th>\n",
       "      <th>765</th>\n",
       "      <th>766</th>\n",
       "      <th>767</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1671</th>\n",
       "      <td>-0.002864</td>\n",
       "      <td>0.217326</td>\n",
       "      <td>0.124349</td>\n",
       "      <td>-0.082548</td>\n",
       "      <td>0.709688</td>\n",
       "      <td>-0.582441</td>\n",
       "      <td>0.257897</td>\n",
       "      <td>0.169356</td>\n",
       "      <td>0.248880</td>\n",
       "      <td>-0.266686</td>\n",
       "      <td>...</td>\n",
       "      <td>0.501767</td>\n",
       "      <td>0.095387</td>\n",
       "      <td>0.340173</td>\n",
       "      <td>0.087452</td>\n",
       "      <td>-0.368359</td>\n",
       "      <td>0.276195</td>\n",
       "      <td>0.238676</td>\n",
       "      <td>-0.219546</td>\n",
       "      <td>0.066603</td>\n",
       "      <td>0.256149</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10951</th>\n",
       "      <td>-0.141048</td>\n",
       "      <td>0.137934</td>\n",
       "      <td>0.131319</td>\n",
       "      <td>0.194773</td>\n",
       "      <td>0.868204</td>\n",
       "      <td>0.078791</td>\n",
       "      <td>-0.131656</td>\n",
       "      <td>0.036244</td>\n",
       "      <td>-0.215749</td>\n",
       "      <td>-0.291946</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.056256</td>\n",
       "      <td>-0.056041</td>\n",
       "      <td>0.147341</td>\n",
       "      <td>0.189665</td>\n",
       "      <td>-0.357366</td>\n",
       "      <td>0.061799</td>\n",
       "      <td>-0.161923</td>\n",
       "      <td>-0.278955</td>\n",
       "      <td>-0.173722</td>\n",
       "      <td>0.065324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5382</th>\n",
       "      <td>-0.252943</td>\n",
       "      <td>0.527507</td>\n",
       "      <td>-0.065608</td>\n",
       "      <td>0.013467</td>\n",
       "      <td>0.207989</td>\n",
       "      <td>0.003881</td>\n",
       "      <td>-0.066281</td>\n",
       "      <td>0.253166</td>\n",
       "      <td>0.021039</td>\n",
       "      <td>0.290957</td>\n",
       "      <td>...</td>\n",
       "      <td>0.180686</td>\n",
       "      <td>-0.042605</td>\n",
       "      <td>-0.173794</td>\n",
       "      <td>-0.079128</td>\n",
       "      <td>-0.169160</td>\n",
       "      <td>0.001316</td>\n",
       "      <td>-0.142593</td>\n",
       "      <td>-0.070816</td>\n",
       "      <td>-0.208826</td>\n",
       "      <td>0.400737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3954</th>\n",
       "      <td>0.054319</td>\n",
       "      <td>0.096738</td>\n",
       "      <td>0.113037</td>\n",
       "      <td>0.032039</td>\n",
       "      <td>0.493064</td>\n",
       "      <td>-0.641102</td>\n",
       "      <td>0.078760</td>\n",
       "      <td>0.402187</td>\n",
       "      <td>0.189743</td>\n",
       "      <td>-0.089538</td>\n",
       "      <td>...</td>\n",
       "      <td>0.123879</td>\n",
       "      <td>-0.285019</td>\n",
       "      <td>-0.297771</td>\n",
       "      <td>0.557171</td>\n",
       "      <td>0.076169</td>\n",
       "      <td>-0.029826</td>\n",
       "      <td>-0.076095</td>\n",
       "      <td>0.225454</td>\n",
       "      <td>0.002135</td>\n",
       "      <td>0.235430</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11193</th>\n",
       "      <td>-0.065858</td>\n",
       "      <td>0.223270</td>\n",
       "      <td>0.507333</td>\n",
       "      <td>0.266193</td>\n",
       "      <td>0.104696</td>\n",
       "      <td>-0.219555</td>\n",
       "      <td>0.146247</td>\n",
       "      <td>0.315649</td>\n",
       "      <td>-0.126193</td>\n",
       "      <td>-0.435461</td>\n",
       "      <td>...</td>\n",
       "      <td>0.163994</td>\n",
       "      <td>0.207813</td>\n",
       "      <td>-0.001871</td>\n",
       "      <td>0.109391</td>\n",
       "      <td>-0.166778</td>\n",
       "      <td>-0.249199</td>\n",
       "      <td>-0.525419</td>\n",
       "      <td>-0.413066</td>\n",
       "      <td>0.119939</td>\n",
       "      <td>0.064297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5861</th>\n",
       "      <td>0.077512</td>\n",
       "      <td>0.322276</td>\n",
       "      <td>0.026697</td>\n",
       "      <td>-0.111393</td>\n",
       "      <td>0.174207</td>\n",
       "      <td>0.235201</td>\n",
       "      <td>0.053888</td>\n",
       "      <td>0.244942</td>\n",
       "      <td>0.181625</td>\n",
       "      <td>-0.226870</td>\n",
       "      <td>...</td>\n",
       "      <td>0.149843</td>\n",
       "      <td>0.311338</td>\n",
       "      <td>0.045975</td>\n",
       "      <td>-0.572319</td>\n",
       "      <td>-0.068256</td>\n",
       "      <td>0.217745</td>\n",
       "      <td>-0.056509</td>\n",
       "      <td>-0.355174</td>\n",
       "      <td>-0.028610</td>\n",
       "      <td>0.090676</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3627</th>\n",
       "      <td>-0.173311</td>\n",
       "      <td>-0.023604</td>\n",
       "      <td>0.190388</td>\n",
       "      <td>-0.136543</td>\n",
       "      <td>-0.360269</td>\n",
       "      <td>-0.444686</td>\n",
       "      <td>0.056311</td>\n",
       "      <td>0.291941</td>\n",
       "      <td>-0.399719</td>\n",
       "      <td>-0.167930</td>\n",
       "      <td>...</td>\n",
       "      <td>0.042209</td>\n",
       "      <td>-0.161905</td>\n",
       "      <td>-0.040535</td>\n",
       "      <td>-0.050515</td>\n",
       "      <td>-0.252020</td>\n",
       "      <td>-0.133980</td>\n",
       "      <td>0.155001</td>\n",
       "      <td>-0.154482</td>\n",
       "      <td>-0.060201</td>\n",
       "      <td>-0.126556</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12559</th>\n",
       "      <td>-0.124635</td>\n",
       "      <td>-0.101799</td>\n",
       "      <td>0.129061</td>\n",
       "      <td>0.636907</td>\n",
       "      <td>0.681090</td>\n",
       "      <td>0.399300</td>\n",
       "      <td>-0.078321</td>\n",
       "      <td>0.221824</td>\n",
       "      <td>-0.277218</td>\n",
       "      <td>-0.178589</td>\n",
       "      <td>...</td>\n",
       "      <td>0.022364</td>\n",
       "      <td>-0.109275</td>\n",
       "      <td>-0.073540</td>\n",
       "      <td>-0.153336</td>\n",
       "      <td>-0.123705</td>\n",
       "      <td>-0.238896</td>\n",
       "      <td>0.296446</td>\n",
       "      <td>-0.116798</td>\n",
       "      <td>0.115076</td>\n",
       "      <td>-0.345925</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8123</th>\n",
       "      <td>0.063508</td>\n",
       "      <td>0.332506</td>\n",
       "      <td>0.119605</td>\n",
       "      <td>-0.001363</td>\n",
       "      <td>-0.161802</td>\n",
       "      <td>-0.082302</td>\n",
       "      <td>-0.025883</td>\n",
       "      <td>0.048027</td>\n",
       "      <td>0.126974</td>\n",
       "      <td>-0.159802</td>\n",
       "      <td>...</td>\n",
       "      <td>0.002221</td>\n",
       "      <td>-0.093885</td>\n",
       "      <td>0.430285</td>\n",
       "      <td>-0.088561</td>\n",
       "      <td>0.321488</td>\n",
       "      <td>0.447437</td>\n",
       "      <td>0.292395</td>\n",
       "      <td>-0.188566</td>\n",
       "      <td>-0.272767</td>\n",
       "      <td>0.126173</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>210</th>\n",
       "      <td>0.015537</td>\n",
       "      <td>0.425568</td>\n",
       "      <td>0.350672</td>\n",
       "      <td>0.113120</td>\n",
       "      <td>-0.128615</td>\n",
       "      <td>0.098112</td>\n",
       "      <td>0.222081</td>\n",
       "      <td>0.101654</td>\n",
       "      <td>0.224073</td>\n",
       "      <td>-0.341074</td>\n",
       "      <td>...</td>\n",
       "      <td>0.100983</td>\n",
       "      <td>-0.008055</td>\n",
       "      <td>0.202025</td>\n",
       "      <td>0.029846</td>\n",
       "      <td>-0.019182</td>\n",
       "      <td>0.107064</td>\n",
       "      <td>0.002301</td>\n",
       "      <td>0.038213</td>\n",
       "      <td>-0.139270</td>\n",
       "      <td>-0.007586</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2928 rows × 768 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            0         1         2         3         4         5         6    \\\n",
       "1671  -0.002864  0.217326  0.124349 -0.082548  0.709688 -0.582441  0.257897   \n",
       "10951 -0.141048  0.137934  0.131319  0.194773  0.868204  0.078791 -0.131656   \n",
       "5382  -0.252943  0.527507 -0.065608  0.013467  0.207989  0.003881 -0.066281   \n",
       "3954   0.054319  0.096738  0.113037  0.032039  0.493064 -0.641102  0.078760   \n",
       "11193 -0.065858  0.223270  0.507333  0.266193  0.104696 -0.219555  0.146247   \n",
       "...         ...       ...       ...       ...       ...       ...       ...   \n",
       "5861   0.077512  0.322276  0.026697 -0.111393  0.174207  0.235201  0.053888   \n",
       "3627  -0.173311 -0.023604  0.190388 -0.136543 -0.360269 -0.444686  0.056311   \n",
       "12559 -0.124635 -0.101799  0.129061  0.636907  0.681090  0.399300 -0.078321   \n",
       "8123   0.063508  0.332506  0.119605 -0.001363 -0.161802 -0.082302 -0.025883   \n",
       "210    0.015537  0.425568  0.350672  0.113120 -0.128615  0.098112  0.222081   \n",
       "\n",
       "            7         8         9    ...       758       759       760  \\\n",
       "1671   0.169356  0.248880 -0.266686  ...  0.501767  0.095387  0.340173   \n",
       "10951  0.036244 -0.215749 -0.291946  ... -0.056256 -0.056041  0.147341   \n",
       "5382   0.253166  0.021039  0.290957  ...  0.180686 -0.042605 -0.173794   \n",
       "3954   0.402187  0.189743 -0.089538  ...  0.123879 -0.285019 -0.297771   \n",
       "11193  0.315649 -0.126193 -0.435461  ...  0.163994  0.207813 -0.001871   \n",
       "...         ...       ...       ...  ...       ...       ...       ...   \n",
       "5861   0.244942  0.181625 -0.226870  ...  0.149843  0.311338  0.045975   \n",
       "3627   0.291941 -0.399719 -0.167930  ...  0.042209 -0.161905 -0.040535   \n",
       "12559  0.221824 -0.277218 -0.178589  ...  0.022364 -0.109275 -0.073540   \n",
       "8123   0.048027  0.126974 -0.159802  ...  0.002221 -0.093885  0.430285   \n",
       "210    0.101654  0.224073 -0.341074  ...  0.100983 -0.008055  0.202025   \n",
       "\n",
       "            761       762       763       764       765       766       767  \n",
       "1671   0.087452 -0.368359  0.276195  0.238676 -0.219546  0.066603  0.256149  \n",
       "10951  0.189665 -0.357366  0.061799 -0.161923 -0.278955 -0.173722  0.065324  \n",
       "5382  -0.079128 -0.169160  0.001316 -0.142593 -0.070816 -0.208826  0.400737  \n",
       "3954   0.557171  0.076169 -0.029826 -0.076095  0.225454  0.002135  0.235430  \n",
       "11193  0.109391 -0.166778 -0.249199 -0.525419 -0.413066  0.119939  0.064297  \n",
       "...         ...       ...       ...       ...       ...       ...       ...  \n",
       "5861  -0.572319 -0.068256  0.217745 -0.056509 -0.355174 -0.028610  0.090676  \n",
       "3627  -0.050515 -0.252020 -0.133980  0.155001 -0.154482 -0.060201 -0.126556  \n",
       "12559 -0.153336 -0.123705 -0.238896  0.296446 -0.116798  0.115076 -0.345925  \n",
       "8123  -0.088561  0.321488  0.447437  0.292395 -0.188566 -0.272767  0.126173  \n",
       "210    0.029846 -0.019182  0.107064  0.002301  0.038213 -0.139270 -0.007586  \n",
       "\n",
       "[2928 rows x 768 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emb_test = embedder.encode(test_df[\"text\"].tolist())\n",
    "emb_test_df = pd.DataFrame(emb_test, index=test_df.index)\n",
    "emb_test_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "b3decaff-0045-4df7-9757-10a6b7341797",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy 0.87\n",
      "Test accuracy 0.83\n"
     ]
    }
   ],
   "source": [
    "lgr = LogisticRegression(max_iter=1000)\n",
    "lgr.fit(emb_train, y_train)\n",
    "print(\"Train accuracy {:.2f}\".format(lgr.score(emb_train, y_train)))\n",
    "print(\"Test accuracy {:.2f}\".format(lgr.score(emb_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c709a6a0-480f-4462-9135-207affa6825f",
   "metadata": {},
   "source": [
    "- Some improvement over bag of words and average embedding representations! \n",
    "- But much slower ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de3930dd-c05d-4c1f-8ea2-53bdfdfa6d1b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### (Optional) Training LDA with [gensim](https://radimrehurek.com/gensim/models/ldamodel.html)\n",
    "\n",
    "- Above we are creating an LDA model with `sklearn`. If you want more flexibility, you can use a Gensim's LDA\n",
    "- To train an LDA model with [gensim](https://radimrehurek.com/gensim/models/ldamodel.html), you need\n",
    "    - Document-term matrix \n",
    "    - Dictionary (vocabulary)\n",
    "    - The number of topics ($K$): `num_topics`\n",
    "    - The number of passes: `passes`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c449d337-860c-4d25-8d82-8883d2747763",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### `Gensim`'s `doc2bow`\n",
    "- Let's first create a dictionary using [`corpora.Dictionary`](https://radimrehurek.com/gensim/corpora/dictionary.html). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "72d4acd2-4061-4eda-9270-4b6d03394ab9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wiki query</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Artificial Intelligence</td>\n",
       "      <td>Artificial intelligence (AI) is the intelligen...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>unsupervised learning</td>\n",
       "      <td>Supervised learning (SL) is a paradigm in mach...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Supreme Court of Canada</td>\n",
       "      <td>The Supreme Court of Canada (SCC; French: Cour...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Peace, Order, and Good Government</td>\n",
       "      <td>In many Commonwealth jurisdictions, the phrase...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Canadian constitutional law</td>\n",
       "      <td>Canadian constitutional law (French: droit con...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ice hockey</td>\n",
       "      <td>Ice hockey (or simply hockey) is a team sport ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          wiki query  \\\n",
       "0            Artificial Intelligence   \n",
       "1              unsupervised learning   \n",
       "2            Supreme Court of Canada   \n",
       "3  Peace, Order, and Good Government   \n",
       "4        Canadian constitutional law   \n",
       "5                         ice hockey   \n",
       "\n",
       "                                                text  \n",
       "0  Artificial intelligence (AI) is the intelligen...  \n",
       "1  Supervised learning (SL) is a paradigm in mach...  \n",
       "2  The Supreme Court of Canada (SCC; French: Cour...  \n",
       "3  In many Commonwealth jurisdictions, the phrase...  \n",
       "4  Canadian constitutional law (French: droit con...  \n",
       "5  Ice hockey (or simply hockey) is a team sport ...  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import wikipedia\n",
    "\n",
    "queries = [\n",
    "    \"Artificial Intelligence\",\n",
    "    \"unsupervised learning\",\n",
    "    \"Supreme Court of Canada\",\n",
    "    \"Peace, Order, and Good Government\",\n",
    "    \"Canadian constitutional law\",\n",
    "    \"ice hockey\",\n",
    "]\n",
    "wiki_dict = {\"wiki query\": [], \"text\": []}\n",
    "for i in range(len(queries)):\n",
    "    wiki_dict[\"text\"].append(wikipedia.page(queries[i]).content)\n",
    "    wiki_dict[\"wiki query\"].append(queries[i])\n",
    "\n",
    "wiki_df = pd.DataFrame(wiki_dict)\n",
    "wiki_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "7e4e2297-d58d-47e3-857f-d0e1bcdfee2e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_md\", disable=[\"parser\", \"ner\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "07f020ea-0c7c-43b9-89b7-149b910e7aa1",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def preprocess(\n",
    "    doc,\n",
    "    min_token_len=2,\n",
    "    irrelevant_pos=[\"ADV\", \"PRON\", \"CCONJ\", \"PUNCT\", \"PART\", \"DET\", \"ADP\", \"SPACE\"],\n",
    "):\n",
    "    \"\"\"\n",
    "    Given text, min_token_len, and irrelevant_pos carry out preprocessing of the text\n",
    "    and return a preprocessed string.\n",
    "\n",
    "    Parameters\n",
    "    -------------\n",
    "    doc : (spaCy doc object)\n",
    "        the spacy doc object of the text\n",
    "    min_token_len : (int)\n",
    "        min_token_length required\n",
    "    irrelevant_pos : (list)\n",
    "        a list of irrelevant pos tags\n",
    "\n",
    "    Returns\n",
    "    -------------\n",
    "    (str) the preprocessed text\n",
    "    \"\"\"\n",
    "\n",
    "    clean_text = []\n",
    "\n",
    "    for token in doc:\n",
    "        if (\n",
    "            token.is_stop == False  # Check if it's not a stopword\n",
    "            and len(token) > min_token_len  # Check if the word meets minimum threshold\n",
    "            and token.pos_ not in irrelevant_pos\n",
    "        ):  # Check if the POS is in the acceptable POS tags\n",
    "            lemma = token.lemma_  # Take the lemma of the word\n",
    "            clean_text.append(lemma.lower())\n",
    "    return \" \".join(clean_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "67d1fa76-6cb7-4108-963d-44b6aea52b2b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "wiki_df[\"text_pp\"] = [preprocess(text) for text in nlp.pipe(wiki_df[\"text\"])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "df0c5045-85d8-4ce1-babf-cbdebf82ea24",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wiki query</th>\n",
       "      <th>text</th>\n",
       "      <th>text_pp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Artificial Intelligence</td>\n",
       "      <td>Artificial intelligence (AI) is the intelligen...</td>\n",
       "      <td>artificial intelligence intelligence machine s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>unsupervised learning</td>\n",
       "      <td>Supervised learning (SL) is a paradigm in mach...</td>\n",
       "      <td>supervised learning paradigm machine learning ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Supreme Court of Canada</td>\n",
       "      <td>The Supreme Court of Canada (SCC; French: Cour...</td>\n",
       "      <td>supreme court canada scc french cour suprême c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Peace, Order, and Good Government</td>\n",
       "      <td>In many Commonwealth jurisdictions, the phrase...</td>\n",
       "      <td>commonwealth jurisdiction phrase peace order g...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Canadian constitutional law</td>\n",
       "      <td>Canadian constitutional law (French: droit con...</td>\n",
       "      <td>canadian constitutional law french droit const...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ice hockey</td>\n",
       "      <td>Ice hockey (or simply hockey) is a team sport ...</td>\n",
       "      <td>ice hockey hockey team sport play ice skate ic...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          wiki query  \\\n",
       "0            Artificial Intelligence   \n",
       "1              unsupervised learning   \n",
       "2            Supreme Court of Canada   \n",
       "3  Peace, Order, and Good Government   \n",
       "4        Canadian constitutional law   \n",
       "5                         ice hockey   \n",
       "\n",
       "                                                text  \\\n",
       "0  Artificial intelligence (AI) is the intelligen...   \n",
       "1  Supervised learning (SL) is a paradigm in mach...   \n",
       "2  The Supreme Court of Canada (SCC; French: Cour...   \n",
       "3  In many Commonwealth jurisdictions, the phrase...   \n",
       "4  Canadian constitutional law (French: droit con...   \n",
       "5  Ice hockey (or simply hockey) is a team sport ...   \n",
       "\n",
       "                                             text_pp  \n",
       "0  artificial intelligence intelligence machine s...  \n",
       "1  supervised learning paradigm machine learning ...  \n",
       "2  supreme court canada scc french cour suprême c...  \n",
       "3  commonwealth jurisdiction phrase peace order g...  \n",
       "4  canadian constitutional law french droit const...  \n",
       "5  ice hockey hockey team sport play ice skate ic...  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wiki_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "0651ba1b-2b0e-4443-8b5a-8411c10407f0",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Word</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\"criticism</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0070087705.these</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0134610993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>127</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1863</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3789</th>\n",
       "      <td>works</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3790</th>\n",
       "      <td>worldwide</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3791</th>\n",
       "      <td>youth</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3792</th>\n",
       "      <td>zhenskaya</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3793</th>\n",
       "      <td>zonal</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3794 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                  Word\n",
       "0           \"criticism\n",
       "1     0070087705.these\n",
       "2           0134610993\n",
       "3                  127\n",
       "4                 1863\n",
       "...                ...\n",
       "3789             works\n",
       "3790         worldwide\n",
       "3791             youth\n",
       "3792         zhenskaya\n",
       "3793             zonal\n",
       "\n",
       "[3794 rows x 1 columns]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import gensim\n",
    "import gensim.corpora as corpora\n",
    "\n",
    "corpus = [doc.split() for doc in wiki_df[\"text_pp\"].tolist()]\n",
    "dictionary = corpora.Dictionary(corpus)  # Create a vocabulary for the lda model\n",
    "pd.DataFrame(\n",
    "    dictionary.token2id.keys(), index=dictionary.token2id.values(), columns=[\"Word\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b6596b6-62c8-44b1-b3d0-ca69a3433629",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### `Gensim`'s `doc2bow`\n",
    "- Now let's convert our corpus into document-term matrix for LDA using [`dictionary.doc2bow`](https://radimrehurek.com/gensim/corpora/dictionary.html#gensim.corpora.dictionary.Dictionary.doc2bow).\n",
    "- For each document, it stores the frequency of each token in the document in the format (token_id, frequency). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "4bfda857-2cbb-4447-a082-5fdd87c51131",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(61, 4),\n",
       " (76, 1),\n",
       " (81, 3),\n",
       " (82, 1),\n",
       " (88, 1),\n",
       " (89, 1),\n",
       " (92, 3),\n",
       " (123, 51),\n",
       " (130, 1),\n",
       " (141, 4),\n",
       " (143, 1),\n",
       " (155, 1),\n",
       " (156, 1),\n",
       " (157, 4),\n",
       " (158, 3),\n",
       " (171, 1),\n",
       " (174, 1),\n",
       " (181, 2),\n",
       " (190, 2),\n",
       " (202, 3)]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc_term_matrix = [dictionary.doc2bow(doc) for doc in corpus]\n",
    "doc_term_matrix[1][:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bda5ffe-699d-48f7-addd-6f35d7144898",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now we are ready to train an LDA model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "48a3cd4b-a67f-4047-be00-f258a262e568",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from gensim.models import LdaModel\n",
    "from gensim.models import LdaModel\n",
    "\n",
    "num_topics = 3\n",
    "\n",
    "lda = gensim.models.LdaModel(\n",
    "    corpus=doc_term_matrix,\n",
    "    id2word=dictionary,\n",
    "    num_topics=num_topics,\n",
    "    random_state=42,\n",
    "    passes=10,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8a2dc90-7b6f-4fce-8886-1d5fc10fdc47",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Examine the topics and topic distribution for a document in our LDA model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "aaef8739-e041-47d4-badd-effdc211e56f",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, '0.031*\"hockey\" + 0.022*\"ice\" + 0.020*\"player\" + 0.019*\"team\"'),\n",
       " (1,\n",
       "  '0.010*\"learning\" + 0.009*\"algorithm\" + 0.009*\"machine\" + 0.009*\"intelligence\"'),\n",
       " (2, '0.029*\"court\" + 0.015*\"law\" + 0.012*\"provincial\" + 0.012*\"government\"')]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda.print_topics(num_words=4)  # Topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "917a8a45-6fa8-483a-9fbd-59ce86190c75",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document:  Artificial Intelligence\n",
      "Topic assignment for document:  [(1, 0.99986523)]\n"
     ]
    }
   ],
   "source": [
    "print(\"Document: \", wiki_df.iloc[0].iloc[0])\n",
    "print(\"Topic assignment for document: \", lda[doc_term_matrix[0]])  # Topic distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4735e79d-4ab8-4bf2-ad6e-9ee4dfcb8b7a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You can also visualize the topics using `pyLDAvis`. \n",
    "\n",
    "```\n",
    "pip install pyLDAvis\n",
    "\n",
    "```\n",
    "\n",
    "> Do not install it using `conda`. They have made some changes in the recent version and `conda` build is not available for this version yet. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d2863fe-2954-42c4-9d32-48a255ac606a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Visualize topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "45d426ec-2cca-4a29-b56c-0001cde8d5d0",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# import pyLDAvis\n",
    "# import pyLDAvis.gensim_models as gensimvis\n",
    "# vis = gensimvis.prepare(lda, doc_term_matrix, dictionary, sort_topics=False)\n",
    "# vis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2da795d-45ee-492e-a12c-8d3a54c21a83",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "cpsc330",
   "language": "python",
   "name": "cpsc330"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}