Covid_Kaggle_n1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Covid : LDA model and a GraphDB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following, I explore my approach used to build an LDA model for topic detection and then ingest the dataset from metadata.csv into a GraphDB. For this purpose, I've used Neo4j which can be easily installed on every computer. I prefer this solution instead of common Python libraries since this is closer to my real job.\n",
    "\n",
    "I'll try to update the results coming from this analysis. At the end, you will find the next and things I want to explore. \n",
    "\n",
    "Thare are already many notebooks on LDA and analysis on metadata file. I want to report the main source of inspiration. These are great notebooks, I suggest to read/learn (and upvote) them.\n",
    "\n",
    "References:\n",
    "* [Topic modeling finding related articles by Daniel Wolffram](https://www.kaggle.com/danielwolffram/topic-modeling-finding-related-articles)\n",
    "* [Tools and Knowledge Graphs by Shahules786](https://www.kaggle.com/shahules/cord-tools-and-knowledge-graphs)\n",
    "* [Topic Modeling with Gensim (web - Tutorial)](https://www.machinelearningplus.com/nlp/topic-modeling-gensim-python/)\n",
    "\n",
    "N.B: this is my first notebook published on Kaggle. Any suggestion is welcome :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T20:34:43.064008Z",
     "start_time": "2020-03-29T20:34:29.103321Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from tqdm import tqdm\n",
    "from functools import reduce\n",
    "import numpy as np\n",
    "import time\n",
    "import re\n",
    "\n",
    "#NLP stuff\n",
    "import unicodedata\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "from nltk.tokenize import ToktokTokenizer\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem.snowball import SnowballStemmer\n",
    "from gensim import corpora\n",
    "import gensim\n",
    "\n",
    "# Visualization\n",
    "import pyLDAvis\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "#Data Ingestion to Neo4j\n",
    "from py2neo import Graph\n",
    "from py2neo import Node,Relationship\n",
    "from py2neo import NodeMatcher\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section I define the functions to clean the text in the following sections. \n",
    "In particular: stemmer, stopwrods, lemmatizer and tokenizer. I plan to test different Tokenizers in the future. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stemmer = SnowballStemmer(\"english\", ignore_stopwords = True)\n",
    "stopWordList=stopwords.words('english')\n",
    "stopWordList.remove('no')\n",
    "stopWordList.remove('not')\n",
    "lemma=WordNetLemmatizer()\n",
    "token=ToktokTokenizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def stopWordsRemove(text):\n",
    "    wordList=[x.lower().strip() for x in token.tokenize(text)]\n",
    "    removedList=[x for x in wordList if not (x in stopWordList)]\n",
    "    text=' '.join(removedList)\n",
    "    return text\n",
    "def stemWords(text):\n",
    "    wordList=[x.lower().strip() for x in token.tokenize(text)]\n",
    "    stemmedlist = map(lambda x: stemmer.stem(x), wordList)\n",
    "    text = ' '.join(stemmedlist)\n",
    "    return text\n",
    "def removeAscendingChar(data):\n",
    "    data=unicodedata.normalize('NFKD',data).encode('ascii', 'ignore').decode('utf-8', 'ignore')\n",
    "    return data\n",
    "def removeCharDigit(text):\n",
    "    stringa='`~@#$%&*()[!{”;:\\’><.,/?”}]0123456789'\n",
    "    text = ''.join(list(map(lambda w: ' ' if w in stringa else w, text)))\n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "source_path = '../Data/'\n",
    "end_path = '../Data/'\n",
    "filename = 'metadata.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(source_path+filename)\n",
    "df['key'] = df.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I had to define a key column to join later the results with the original dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "papers = df[['title','abstract','key']]\n",
    "papers.abstract = papers.abstract.str.replace('Abstract ','')\n",
    "papers = papers.dropna(how='all',subset=['title'])\n",
    "papers = papers.fillna(value=' ')\n",
    "papers = papers.drop_duplicates(['title','abstract'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "columns = ['title','abstract']\n",
    "\n",
    "for column in tqdm(columns):\n",
    "    papers[column] = papers[column].apply(lambda x: removeAscendingChar(x))\n",
    "    papers[column] = papers[column].apply(lambda x: removeCharDigit(str(x)))\n",
    "    papers[column] = papers[column].apply(lambda x: stemWords(stopWordsRemove(str(x))))\n",
    "papers['text'] = papers.title + ' ' + papers.abstract"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T20:38:28.352105Z",
     "start_time": "2020-03-29T20:38:28.348629Z"
    }
   },
   "source": [
    "### Build Corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_corpus(df,column):\n",
    "    corpus = []\n",
    "    for data in tqdm(df[column].dropna()):\n",
    "        words = [x for x in token.tokenize(data) if not (x in stopWordList)]\n",
    "        words = [lemma.lemmatize(x) for x in words if len(x) > 2 ]\n",
    "        corpus.append(words)\n",
    "    return corpus   \n",
    "def prepare_text(text):\n",
    "    words = [x for x in token.tokenize(removeAscendingChar(str(text))) if not (x in stopWordList)]\n",
    "    words = [lemma.lemmatize(x) for x in words if len(x) > 2 ]\n",
    "    return words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_corpus = build_corpus(papers,'text')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dictionary = corpora.Dictionary(text_corpus)\n",
    "dictionary.filter_extremes(no_below = 3, no_above = 0.99)\n",
    "dictionary.save('../Model/dictionary')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save Dictionary for later use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = [dictionary.doc2bow(text) for text in text_corpus]\n",
    "papers = pd.concat([papers,pd.Series(text_corpus,name = 'text_corpus')], axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Find Optimal Number of Topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models import CoherenceModel,LdaModel,HdpModel\n",
    "def compute_coherence_values(dictionary, corpus, texts, limit, start=2, step=3):\n",
    "    coherence_values = []\n",
    "    model_list = []\n",
    "    for num_topics in tqdm(range(start, limit, step)):\n",
    "        model = LdaModel(corpus=corpus, num_topics=num_topics, id2word=dictionary)\n",
    "        model_list.append(model)\n",
    "        coherencemodel = CoherenceModel(model=model, texts=texts, dictionary=dictionary, coherence='c_v')\n",
    "        coherence_values.append(coherencemodel.get_coherence())\n",
    "\n",
    "    return model_list, coherence_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_best_model(models, score_list, numbers):\n",
    "    m = max(score_list)\n",
    "    k = min([i for i,j in enumerate(score_list) if j==m])\n",
    "    print(\"The best model is number:\", k)\n",
    "    return numbers[k],models[k]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "LIMIT_NUM = 20\n",
    "STEP_NUM = 3\n",
    "START_NUM = 7\n",
    "model_list, coherence_values = compute_coherence_values(dictionary=dictionary,\n",
    "                                                        corpus=corpus,\n",
    "                                                        texts=papers['text_corpus'].fillna(''), \n",
    "                                                        start=START_NUM, \n",
    "                                                        limit=LIMIT_NUM, \n",
    "                                                        step=STEP_NUM)\n",
    "# Show graph\n",
    "limit=LIMIT_NUM; start=START_NUM; step=STEP_NUM;\n",
    "x = range(start, limit, step)\n",
    "plt.plot(x, coherence_values)\n",
    "plt.xlabel(\"Num Topics\")\n",
    "plt.ylabel(\"Coherence score\")\n",
    "plt.legend((\"coherence_values\"), loc='best')\n",
    "plt.show()\n",
    "# Print the coherence scores\n",
    "for m, cv in zip(x, coherence_values):\n",
    "    print(\"Num Topics =\", m, \" has Coherence Value of\", round(cv, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In my case, it seems that 19 is the optimal number of topics. I could increase LIMIT_NUM in the future. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_topics, lda_model = get_best_model(model_list, coherence_values,x)\n",
    "lda_model.save('../Model/lda.model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LDA model Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyLDAvis import gensim\n",
    "lda_display = gensim.prepare(ldamodel3, corpus, dictionary, sort_topics=False)\n",
    "pyLDAvis.show(lda_display)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate Topic Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def topics_sentences(model, corpus, dataset):\n",
    "    \"\"\"\n",
    "   Function to get:\n",
    "   - percentage per topic for every document in texts\n",
    "   - dominant topic \n",
    "   - keywords \n",
    "    \"\"\"\n",
    "    contents = dataset[['title','text_corpus','key']]\n",
    "    df = pd.DataFrame()\n",
    "    topics_text = model[corpus]\n",
    "    for i,row in tqdm(enumerate(topics_text)):\n",
    "        row = sorted(row, key = lambda x: x[1], reverse = True)\n",
    "        for j, (topic_num, prob_topic) in enumerate(row):\n",
    "            if j==0:\n",
    "                wp = model.show_topic(topic_num)\n",
    "                topic_keywords = \", \".join([word for word, prop in wp])\n",
    "                df = df.append(\n",
    "                    pd.Series([int(topic_num),round(prob_topic,4),topic_keywords]),\n",
    "                    ignore_index = True)\n",
    "            else:\n",
    "                break\n",
    "    df = pd.concat([df, contents], axis = 1).reset_index()\n",
    "    df.columns = ['Document_Number','Dominant_Topic','Perc_Contribution', 'Topic_Keywords','title', 'text','key']\n",
    "    df.info()\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_per_document = topics_sentences(lda_model, corpus, papers)\n",
    "topic_per_document.info()\n",
    "topic_per_document.to_csv(end_path+'topic_per_document.csv',header = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topic_distribution(model, corpus):\n",
    "    columns = ['Topic_no_'+str(x) for x in range(0,num_topics)]\n",
    "    topic_distr_docs = pd.DataFrame()\n",
    "    for i,document in tqdm(enumerate(corpus)):\n",
    "        topic_distr_docs = topic_distr_docs.append(pd.Series(list(list(\n",
    "            zip(*model.get_document_topics(document, minimum_probability=0)))[1])\n",
    "                                                             ,dtype = 'float64'),\n",
    "                                                   ignore_index = True\n",
    "                                                  )\n",
    "    topic_distr_docs.columns=columns\n",
    "    return topic_distr_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_distr_docs = get_topic_distribution(lda_model, corpus)\n",
    "topic_distr_docs.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "papers_fe = pd.concat([papers.dropna(subset=['title','abstract']),topic_per_document.dropna(subset=['Dominant_Topic']).drop(['title'],axis = 1),topic_distr_docs],axis = 1).dropna(subset=['text','Dominant_Topic'],how = 'any',axis = 0)\n",
    "papers_fe = papers_fe.loc[:,~papers_fe.columns.duplicated()]\n",
    "papers_fe.to_csv(end_path + 'papers_text.csv',header = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Store Topic Infos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Topics_count = papers_fe['Dominant_Topic'].value_counts()\n",
    "Topic_perc = Topics_count/Topics_count.sum()\n",
    "Topic_Keywords = papers_fe[['Dominant_Topic','Topic_Keywords']].drop_duplicates().set_index('Dominant_Topic')\n",
    "#pd.concat([Topic_Keywords,Topic_perc,Topics_count],axis = 1)\n",
    "Topics = Topic_Keywords.join(pd.concat([Topics_count,Topic_perc],axis=1))\n",
    "Topics.columns=['Topic_Keywords','Topic_Count','Topic_perc']\n",
    "Topics.to_csv('../Data/topics.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GraphDB: Ingestion to Neoj"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "source_path = '../Data/'\n",
    "end_path = '../Data/'\n",
    "filename = 'metadata.csv'\n",
    "df = pd.read_csv(source_path+filename)\n",
    "df['key'] = df.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next lines I add the information previously elaborated. \n",
    "I think there are (at least to way) two ways:\n",
    "* join the data with the original dataset and store it in the DB (this is the method I've chosen at the moment but I still need to study the execution time).\n",
    "* load Dictionary and the LDA model to find the features for each row in the original dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "documents = pd.read_csv('../Data/papers_text.csv',index_col = 0)\n",
    "documents.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_document = df.join(documents,on='key',how='inner',lsuffix='_orig')\n",
    "full_document['authors'] = full_document['authors'].apply(lambda x: removeAscendingChar(str(x)).split(';'))\n",
    "full_document.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Topics = pd.read_csv('../Data/topics.csv')\n",
    "Topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to declare the connection to our graphdb istance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "remote_graph = Graph(\"bolt://localhost:<YOUR-LOCAL-HOST>\",user = \"neo4j\",password = \"<YOUR-PASSWORD>\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create the nodes for the topics with the uniqueness constrain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "remote_graph.schema.create_uniqueness_constraint('Topic','id_topic')\n",
    "\n",
    "for index,row in tqdm(Topics.iterrows()):\n",
    "    a = Node('Topic', id_topic = row.Dominant_Topic, keywords = row.Topic_Keywords)\n",
    "    a.__primarylabel__ = 'Topic'\n",
    "    a.__primarykey__ = 'id_topic'\n",
    "    remote_graph.create(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define some constraint for the next entities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    remote_graph.schema.create_uniqueness_constraint('Paper','title')\n",
    "except:\n",
    "    pass\n",
    "try:\n",
    "    remote_graph.schema.create_uniqueness_constraint('Author','name')\n",
    "except:\n",
    "    pass\n",
    "try:\n",
    "    remote_graph.schema.create_uniqueness_constraint('Journal','name')\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stage = full_document\n",
    "\n",
    "cols = list(stage.columns)\n",
    "r = re.compile('Topic_no.*')\n",
    "topic_columns = list(filter(r.match,cols))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "AUTHORED_BY = Relationship.type(\"AUTHORED_BY\")\n",
    "PUBLISHED_IN = Relationship.type(\"PUBLISHED_IN\")\n",
    "DOMINANT_TOPIC = Relationship.type(\"DOMINANT_TOPIC\")\n",
    "\n",
    "matcher = NodeMatcher(remote_graph)\n",
    "\n",
    "for index, row in tqdm(stage.iterrows()):\n",
    "    a = Node(\"Paper\", title = row.title, doi = row.doi, abstract = row.abstract, pmcid = row.pmcid, sha = row.sha,\n",
    "             license = row.license, pusblish_date = row.publish_time,\n",
    "             has_full_text = row.has_full_text, full_text_file = row.full_text_file             \n",
    "            )\n",
    "    a.__primarylabel__ = \"Paper\"\n",
    "    a.__primarykey__ = \"title\"\n",
    "    for author in row.authors:\n",
    "        b = Node(\"Author\", name = author)\n",
    "        b.__primarylabel__ = \"Author\"\n",
    "        b.__primarykey__ = \"name\"\n",
    "        remote_graph.merge(AUTHORED_BY(a, b))\n",
    "    c = Node(\"Journal\", name = row.journal, source_x = row.source_x)\n",
    "    c.__primarylabel__=\"Journal\"\n",
    "    c.__primarykey__=\"name\"\n",
    "    remote_graph.merge(PUBLISHED_IN(a,c))\n",
    "    scores = []\n",
    "    for topic in topic_columns:\n",
    "        scores.append(row[topic])\n",
    "    a['topic_score'] = scores\n",
    "    remote_graph.push(a)\n",
    "    d = matcher.match(\"Topic\", id_topic = row.Dominant_Topic).first()\n",
    "    remote_graph.merge(DOMINANT_TOPIC(a,d))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}