PPCP_soil_papers_2000-2022.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5711487e-c6e0-4cde-9f47-c077a612b54f",
   "metadata": {
    "id": "5711487e-c6e0-4cde-9f47-c077a612b54f"
   },
   "source": [
    "# Project description\n",
    "\n",
    "This notebook represents data analytics on the topic of research dedicated to the spread of emerging contaminants -- pharmaceuticals and products of personal care products (PPCP). Influence of PPCP is mostly studied with the focus on the aquatic environment, however it is also worth to know the current state of the research dedicated to the less studied soil environment. To achieve this, analysis of titles and abstracts of peer-reviwed research papers published from 2000 to 2022 is performed.\n",
    "\n",
    "This notebook consists of:\n",
    "* data acquisition \n",
    "* data preparation\n",
    "* data comparison\n",
    "* and data visualisation steps.\n",
    "\n",
    "Each step is supplied with comments, so it would be easy to understand the analysis workflow and reproduce it.\n",
    "To perform analysis data from Dimensions scientific database https://www.dimensions.ai and Umwelt Bundesamt (UBA) Pharmaceuticals in the environment https://www.umweltbundesamt.de/en/database-pharmaceuticals-in-the-environment-0 were used."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e1ea36b-0e2d-4dcf-89dd-6c6f4a25ec27",
   "metadata": {
    "id": "2e1ea36b-0e2d-4dcf-89dd-6c6f4a25ec27"
   },
   "source": [
    "## Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8c39bafb-dad0-45bf-96a2-b904bbbb9a6b",
   "metadata": {
    "id": "8c39bafb-dad0-45bf-96a2-b904bbbb9a6b"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import math\n",
    "import regex as re\n",
    "import numpy as np\n",
    "import os\n",
    "import glob\n",
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0cbee90-0552-4656-a6e9-6fe844ccb311",
   "metadata": {
    "id": "d0cbee90-0552-4656-a6e9-6fe844ccb311"
   },
   "source": [
    "## Creating peer-reviewed papers work dataset "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "y-qXnBuWRmS0",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "y-qXnBuWRmS0",
    "outputId": "ac0c8ce3-8ad9-418f-9cdc-8dc0f2f2423c"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "  import google.colab\n",
    "  !git clone https://github.com/po1line/Analytics-on-research-of-pharmaceuticals-entry-in-soils\n",
    "  %cd Analytics-on-research-of-pharmaceuticals-entry-in-soils\n",
    "except Exception:\n",
    "  pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "46a71ca9-b1bd-4d04-978f-628963361cf0",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "46a71ca9-b1bd-4d04-978f-628963361cf0",
    "outputId": "b3331ce1-9b17-49d8-dfe2-003049fbdce7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "papers_2007-2000.xlsx papers_2017-2016.xlsx papers_2020.xlsx\n",
      "papers_2012-2008.xlsx papers_2018.xlsx      papers_2021.xlsx\n",
      "papers_2015-2013.xlsx papers_2019.xlsx      papers_2022.xlsx\n"
     ]
    }
   ],
   "source": [
    "!ls data/dimensions/\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c7ff2cac-4a52-41e4-aa5f-b4dbd2d729df",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "c7ff2cac-4a52-41e4-aa5f-b4dbd2d729df",
    "outputId": "529f1639-a2cd-48d2-ec90-12c3f643f936"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['data/dimensions/papers_2012-2008.xlsx',\n",
      " 'data/dimensions/papers_2021.xlsx',\n",
      " 'data/dimensions/papers_2020.xlsx',\n",
      " 'data/dimensions/papers_2015-2013.xlsx',\n",
      " 'data/dimensions/papers_2017-2016.xlsx',\n",
      " 'data/dimensions/papers_2007-2000.xlsx',\n",
      " 'data/dimensions/papers_2019.xlsx',\n",
      " 'data/dimensions/papers_2022.xlsx',\n",
      " 'data/dimensions/papers_2018.xlsx']\n"
     ]
    }
   ],
   "source": [
    "#folder\n",
    "path = \"data/dimensions/\"\n",
    "#global search in set folder of xlsx files\n",
    "filenames = glob.glob(path + \"*.xlsx\")\n",
    "#looking at the files\n",
    "pprint(filenames)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d74f2485-ba5b-4bc4-a9ba-3f3ab7e72e05",
   "metadata": {
    "id": "d74f2485-ba5b-4bc4-a9ba-3f3ab7e72e05"
   },
   "outputs": [],
   "source": [
    "#collecting files in merged dataframe\n",
    "df = pd.DataFrame() #starting with empty dataframe\n",
    "for file_name in filenames:\n",
    "    data = pd.read_excel(file_name, \n",
    "                         skiprows=[0])\n",
    "    df = pd.concat((df,data)).reset_index(drop=True)\n",
    "#appending each read table in the empty dataframe\n",
    "#important! do not forget to drop index at appending stage to avoid multiple index search results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "10392b6d-27ba-4b64-9123-5337c095a6e4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "10392b6d-27ba-4b64-9123-5337c095a6e4",
    "outputId": "dede9aa4-7017-4218-cfff-94ed5b88e4fd"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Rank</th>\n",
       "      <th>Publication ID</th>\n",
       "      <th>DOI</th>\n",
       "      <th>PMID</th>\n",
       "      <th>PMCID</th>\n",
       "      <th>Title</th>\n",
       "      <th>Abstract</th>\n",
       "      <th>Acknowledgements</th>\n",
       "      <th>Source title</th>\n",
       "      <th>Anthology title</th>\n",
       "      <th>...</th>\n",
       "      <th>Corresponding Authors</th>\n",
       "      <th>Authors Affiliations</th>\n",
       "      <th>Times cited</th>\n",
       "      <th>Recent citations</th>\n",
       "      <th>RCR</th>\n",
       "      <th>FCR</th>\n",
       "      <th>Source Linkout</th>\n",
       "      <th>Dimensions URL</th>\n",
       "      <th>FOR (ANZSRC) Categories</th>\n",
       "      <th>Sustainable Development Goals</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3405</td>\n",
       "      <td>pub.1034753825</td>\n",
       "      <td>10.1002/etc.487</td>\n",
       "      <td>21312249.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Influence of sewage and pharmaceuticals on soi...</td>\n",
       "      <td>Although sewage effluent application to land i...</td>\n",
       "      <td>We thank Catriona MacDonald and ESR for advice...</td>\n",
       "      <td>Environmental Toxicology and Chemistry</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Gielen, Gerty J. H. P. (Scion); Clinton, Peter...</td>\n",
       "      <td>14</td>\n",
       "      <td>3</td>\n",
       "      <td>0.43</td>\n",
       "      <td>1.47</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://app.dimensions.ai/details/publication/...</td>\n",
       "      <td>0605 Microbiology; 0503 Soil Sciences; 05 Envi...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3376</td>\n",
       "      <td>pub.1069010127</td>\n",
       "      <td>10.2134/jeq2009.0090</td>\n",
       "      <td>20830908.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Selected Veterinary Pharmaceuticals in Agricul...</td>\n",
       "      <td>Veterinary pharmaceuticals are commonly admini...</td>\n",
       "      <td>This study was supported in part by Agricultur...</td>\n",
       "      <td>Journal of Environmental Quality</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Li, Hui (Michigan State University)</td>\n",
       "      <td>Song, Wenlu (Michigan State University); Ding,...</td>\n",
       "      <td>48</td>\n",
       "      <td>3</td>\n",
       "      <td>1.74</td>\n",
       "      <td>4.91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://app.dimensions.ai/details/publication/...</td>\n",
       "      <td>0503 Soil Sciences; 05 Environmental Sciences</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3320</td>\n",
       "      <td>pub.1026757075</td>\n",
       "      <td>10.1007/s00244-010-9500-y</td>\n",
       "      <td>20340015.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Dissipation and Leaching Potential of Selected...</td>\n",
       "      <td>Biosolids land application is an important pat...</td>\n",
       "      <td>This research was funded by the USDA-CSREES pr...</td>\n",
       "      <td>Archives of Environmental Contamination and To...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Wu, Chenxi (University of Toledo)</td>\n",
       "      <td>Wu, Chenxi (University of Toledo); Spongberg, ...</td>\n",
       "      <td>38</td>\n",
       "      <td>4</td>\n",
       "      <td>1.20</td>\n",
       "      <td>3.97</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://app.dimensions.ai/details/publication/...</td>\n",
       "      <td>0503 Soil Sciences; 05 Environmental Sciences</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Rank  Publication ID                        DOI        PMID PMCID  \\\n",
       "0  3405  pub.1034753825            10.1002/etc.487  21312249.0   NaN   \n",
       "1  3376  pub.1069010127       10.2134/jeq2009.0090  20830908.0   NaN   \n",
       "2  3320  pub.1026757075  10.1007/s00244-010-9500-y  20340015.0   NaN   \n",
       "\n",
       "                                               Title  \\\n",
       "0  Influence of sewage and pharmaceuticals on soi...   \n",
       "1  Selected Veterinary Pharmaceuticals in Agricul...   \n",
       "2  Dissipation and Leaching Potential of Selected...   \n",
       "\n",
       "                                            Abstract  \\\n",
       "0  Although sewage effluent application to land i...   \n",
       "1  Veterinary pharmaceuticals are commonly admini...   \n",
       "2  Biosolids land application is an important pat...   \n",
       "\n",
       "                                    Acknowledgements  \\\n",
       "0  We thank Catriona MacDonald and ESR for advice...   \n",
       "1  This study was supported in part by Agricultur...   \n",
       "2  This research was funded by the USDA-CSREES pr...   \n",
       "\n",
       "                                        Source title Anthology title  ...  \\\n",
       "0             Environmental Toxicology and Chemistry             NaN  ...   \n",
       "1                   Journal of Environmental Quality             NaN  ...   \n",
       "2  Archives of Environmental Contamination and To...             NaN  ...   \n",
       "\n",
       "                 Corresponding Authors  \\\n",
       "0                                  NaN   \n",
       "1  Li, Hui (Michigan State University)   \n",
       "2    Wu, Chenxi (University of Toledo)   \n",
       "\n",
       "                                Authors Affiliations  Times cited  \\\n",
       "0  Gielen, Gerty J. H. P. (Scion); Clinton, Peter...           14   \n",
       "1  Song, Wenlu (Michigan State University); Ding,...           48   \n",
       "2  Wu, Chenxi (University of Toledo); Spongberg, ...           38   \n",
       "\n",
       "  Recent citations   RCR   FCR Source Linkout  \\\n",
       "0                3  0.43  1.47            NaN   \n",
       "1                3  1.74  4.91            NaN   \n",
       "2                4  1.20  3.97            NaN   \n",
       "\n",
       "                                      Dimensions URL  \\\n",
       "0  https://app.dimensions.ai/details/publication/...   \n",
       "1  https://app.dimensions.ai/details/publication/...   \n",
       "2  https://app.dimensions.ai/details/publication/...   \n",
       "\n",
       "                             FOR (ANZSRC) Categories  \\\n",
       "0  0605 Microbiology; 0503 Soil Sciences; 05 Envi...   \n",
       "1      0503 Soil Sciences; 05 Environmental Sciences   \n",
       "2      0503 Soil Sciences; 05 Environmental Sciences   \n",
       "\n",
       "  Sustainable Development Goals  \n",
       "0                           NaN  \n",
       "1                           NaN  \n",
       "2                           NaN  \n",
       "\n",
       "[3 rows x 32 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 2983 entries, 0 to 2982\n",
      "Data columns (total 32 columns):\n",
      " #   Column                         Non-Null Count  Dtype  \n",
      "---  ------                         --------------  -----  \n",
      " 0   Rank                           2983 non-null   int64  \n",
      " 1   Publication ID                 2983 non-null   object \n",
      " 2   DOI                            2950 non-null   object \n",
      " 3   PMID                           1432 non-null   float64\n",
      " 4   PMCID                          317 non-null    object \n",
      " 5   Title                          2983 non-null   object \n",
      " 6   Abstract                       2948 non-null   object \n",
      " 7   Acknowledgements               1518 non-null   object \n",
      " 8   Source title                   2684 non-null   object \n",
      " 9   Anthology title                471 non-null    object \n",
      " 10  MeSH terms                     1224 non-null   object \n",
      " 11  Publication Date               2983 non-null   object \n",
      " 12  PubYear                        2983 non-null   int64  \n",
      " 13  Publication Date (online)      2298 non-null   object \n",
      " 14  Publication Date (print)       2516 non-null   object \n",
      " 15  Volume                         2517 non-null   object \n",
      " 16  Issue                          1797 non-null   object \n",
      " 17  Pagination                     2893 non-null   object \n",
      " 18  Open Access                    2983 non-null   object \n",
      " 19  Publication Type               2983 non-null   object \n",
      " 20  Authors                        2968 non-null   object \n",
      " 21  Authors (Raw Affiliation)      2968 non-null   object \n",
      " 22  Corresponding Authors          1914 non-null   object \n",
      " 23  Authors Affiliations           2968 non-null   object \n",
      " 24  Times cited                    2983 non-null   int64  \n",
      " 25  Recent citations               2983 non-null   int64  \n",
      " 26  RCR                            1122 non-null   float64\n",
      " 27  FCR                            2171 non-null   float64\n",
      " 28  Source Linkout                 1073 non-null   object \n",
      " 29  Dimensions URL                 2983 non-null   object \n",
      " 30  FOR (ANZSRC) Categories        2945 non-null   object \n",
      " 31  Sustainable Development Goals  365 non-null    object \n",
      "dtypes: float64(3), int64(4), object(25)\n",
      "memory usage: 745.9+ KB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "#look at the data\n",
    "display(df.head(3))\n",
    "print(df.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0917ad90-48a8-44b2-89ba-f1fb909d6e4f",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0917ad90-48a8-44b2-89ba-f1fb909d6e4f",
    "outputId": "cae34f52-8d8a-4c52-f007-192c83e860ae"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 2951 entries, 0 to 2950\n",
      "Data columns (total 32 columns):\n",
      " #   Column                         Non-Null Count  Dtype  \n",
      "---  ------                         --------------  -----  \n",
      " 0   Rank                           2951 non-null   int64  \n",
      " 1   Publication ID                 2951 non-null   object \n",
      " 2   DOI                            2950 non-null   object \n",
      " 3   PMID                           1400 non-null   float64\n",
      " 4   PMCID                          312 non-null    object \n",
      " 5   Title                          2951 non-null   object \n",
      " 6   Abstract                       2918 non-null   object \n",
      " 7   Acknowledgements               1515 non-null   object \n",
      " 8   Source title                   2652 non-null   object \n",
      " 9   Anthology title                471 non-null    object \n",
      " 10  MeSH terms                     1195 non-null   object \n",
      " 11  Publication Date               2951 non-null   object \n",
      " 12  PubYear                        2951 non-null   int64  \n",
      " 13  Publication Date (online)      2298 non-null   object \n",
      " 14  Publication Date (print)       2484 non-null   object \n",
      " 15  Volume                         2487 non-null   object \n",
      " 16  Issue                          1767 non-null   object \n",
      " 17  Pagination                     2861 non-null   object \n",
      " 18  Open Access                    2951 non-null   object \n",
      " 19  Publication Type               2951 non-null   object \n",
      " 20  Authors                        2937 non-null   object \n",
      " 21  Authors (Raw Affiliation)      2937 non-null   object \n",
      " 22  Corresponding Authors          1914 non-null   object \n",
      " 23  Authors Affiliations           2937 non-null   object \n",
      " 24  Times cited                    2951 non-null   int64  \n",
      " 25  Recent citations               2951 non-null   int64  \n",
      " 26  RCR                            1100 non-null   float64\n",
      " 27  FCR                            2141 non-null   float64\n",
      " 28  Source Linkout                 1073 non-null   object \n",
      " 29  Dimensions URL                 2951 non-null   object \n",
      " 30  FOR (ANZSRC) Categories        2914 non-null   object \n",
      " 31  Sustainable Development Goals  362 non-null    object \n",
      "dtypes: float64(3), int64(4), object(25)\n",
      "memory usage: 737.9+ KB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# removing duplicated entries with the same DOIs  \n",
    "df.drop_duplicates(subset = ['DOI'], inplace=True, ignore_index=True)\n",
    "print(df.info())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6c2c45e-5ea8-4340-bc11-4ef4de23f627",
   "metadata": {
    "id": "c6c2c45e-5ea8-4340-bc11-4ef4de23f627",
    "tags": []
   },
   "source": [
    "### First look at the publication statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2fadba2f-70b8-44f8-994b-6a9cde053aa2",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 432
    },
    "id": "2fadba2f-70b8-44f8-994b-6a9cde053aa2",
    "outputId": "5249edc3-3bd7-4f2e-b8ef-2faea75212e1"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAu8AAAGfCAYAAADvSTwNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAABAKUlEQVR4nO3dd5gsZZ238fsLEiSJhAVREVAUxMxRAZEgYMRVzKsuigEVc44oRnDNiVVcFdE1v4oJlSzionIQAyKIgqLEg+Qsh9/7R9WRZpyZMz2nu6tn5v5cV10zXU91fZ+eM6f61zVPPZWqQpIkSdL4W6nrDkiSJEmaGYt3SZIkaY6weJckSZLmCIt3SZIkaY6weJckSZLmCIt3SZIkaY6weJckSZLmCIt3SZIkaY64TdcdAEhye+DfgAKWVNVlHXdJkiRJGjvp4g6rSVYCHgc8GdgZ2HjCJhcCxwNfB75TVTePtIOSJEnSGBpp8Z5kZeBFwBuATYBrgJOBPwF/BwKsB9wNWASsCVwAvAf4ZFUtHVlnJUmSpDEz6uL9TOBOwFeALwAnTHVWvT07vwvwn8BTgHOrausRdVWSJEkaO6Mu3j8IvLeqLurzeRsDr6+qVw6nZ5IkSdL462TMuyRJkqT+OVWkJEmSNEdYvEuSJElzRCfzvCfZCngHcC/gIuDQqvr8JNs9AzisqlYecRcntcEGG9Rmm23WdTckSZI0j51yyimXVNWGk7WNvHhPcifgJOB2wKXAXYGdkjwZeFpVXT3qPs3UZpttxuLFi7vuhiRJkuaxJH+Zqq2LYTP7A6sCe1TVBsCGwAeARwFHJ7ldB32SJEmSxl4XxfvONDdcOgagqq6sqtcBTwXuR1PAr9tBvyRJkqSx1kXxvilw2sSVVfUN4Ak04+CPSXL7UXdMkiRJGmddFO+XABtM1lBVRwB7AfcEjgHWH2G/JEmSpLHWRfH+e+BhUzVW1Q9pCvitgING1SlJkiRp3HVRvB8B7J5k86k2aAv4xwMZVackSZKkcdfFPO+fB84Alk63UVUdmWRHmjHwkiRJ0oI38uK9qi4HfjTDbU8BThlqhyRJkqQ5oothM5IkSZJmobPiPck9kzwlyc5JVp5im3sneeuo+yZJkiSNo06K9ySfAn4LfBk4Fvh9kh0m2fQ+wNtG2TdJkiRpXI28eE/yNOD5wE+AlwPvp5n3/bgk/zHq/kiSJElzRRezzbwE+FlV7bJsRZKPAN8CDkuySlUd1kG/JEmSpLHWxbCZuwNf7V1RVecDuwDHAZ9Nsk8H/ZIkSZLGWhdn3lcDrpm4sqquS/JY4NvAp5OsBFw/6s5JkiRp7klmf2/PqhpgT4ari+L9L8C9J2uoqhuSPI6mgP8UcMwoOyZJkiSNsy6GzfwE2CtTfDyqqhuAxwFHAXuMsmOSJEnSOOuieP8qcB6w41Qb9BTw3wTOHVG/JEmSpLE28mEzVXUCsP0MtrsReNLweyRJkiTNDZ3dYVWSJElSf7q4YPWfkiwCHgjcEVgDuJZmSM3iqjq5y75JkiRJ46aT4j3Jw4BP0Mz5PtmFq5XkD8BLq+rokXZOkiRJGlMjL96T7Ar8EDgfeBvwM5qz7dcDq9Ochd8eeB5wRJJHVNVxo+6nJEmSNG66GPP+duC3wDZV9a6qOrqqfl9V57Rfj66qdwLbAL8DDuhn50l2SvKdJOclqSTP7mlbJcl7k/wmyTVJLkjypSSbDvD1SZIkSUPRRfH+AODTVfUvd1ntVVVXA4cA2/a5/7WA04CXA9dNaFujzX93+/VxwJ2BHybpdPy/JEmStDxdFKw3AuvMcNt12u1nrKqOAI4ASHLohLYrmHDjpyQvoDnDvzXNXwQkSZKksdTFmfcTgFcmufd0G7XtrwR+POT+LPsgcdmQcyRJkqQV0sWZ99cBJwG/THIct1ywegOwGrdcsLoLcAXwhmF1JMmqwAeA71bV36bYZl9gX4BNN3VovCRJkrrTxR1W/5DkgcBBwGOB3SfZ7AbgW8Abq+pPw+hHO8b9i8C6wL9PtV1VHUIz9p5FixbVMPoiSZIkzUQnF2lW1dnAU5KsAdwH2IRbbtJ0PvDb5V3QuiLawv3LwL2BXarq78PKkiRJkgal0xlWqupammEzI5NkFeArwL1oCvcLR5kvSZIkzda8mx4xyVrA3dqHKwGbJrkfcCnNWf2vAw+kGbJTSTZut72iqiZOLSlJkiSNjS5mmyHJakn2S3Jwkrcl2XKK7XZPcmyfu18EnNout6W5KdSpwDuAO9HM7b4JcApwQc/y1Fm9GEmSJGlERn7mvR3n/lOase5pV78pyf5V9V8TNt8I2Lmf/VfV8T37nbQL/exPkiRJGhddnHl/BXBf4D00BfxjaM6CH5jk4A76I0mSJM0JXYx5fzLw1arav318WpIfAR8BXpxklap6fgf9kiRJksZaF8X7XYH/7l1RVTcDL01yOfDmJCtV1XM76JskSZI0troo3q8HVpmsoar2T7IUeGuSlYB+L1aVJEmS5q0uivc/AdsBn5issaoOSALwVmCX0XVLkiRJGm9dXLB6FPDvSdaeaoOqOgA4ALjLiPokSZIkjb0uzrx/gWbYzJbAL6faqKrekeTvNPO2S5IkSQveyIv3qjoLeOMMt510aI0kSZK0EHVyh9WpJFkpyaZJVu26L5IkSdK4GaviHdgQOAfYseuOSJIkSeNm3Ip3gHTdAUmSJGkcjWPxLkmSJGkSFu+SJEnSHNHFVJHTuQ74PHB+1x2RJEnS7LQ33JyVqhpgT+afsSreq+pKYJ+u+yFJkiSNo06L9yQrA3cH7gisAVwLnAf8oaqWdtk3SZIkadx0Urwn2Rg4AHgqsM4km1yZ5GvAAVV1wSj7JkmSJI2rkRfvSTYDTgQ2Bo4HfkZztv16YHWas/DbA88F9kyyY1WdM+p+SpIkSeOmizPv/9XmbltVv55qoyT3BX4EHERzhl6SJEla0LqYKnI34EPTFe4AbfuHgd1H0SlJkiRp3HVRvK8GXDHDba9ot5ckSZIWvC6K918Dz0ty2+k2SrIG8HzgNyPplSRJkjTmuhjz/m7gu8DvkvwPt1ywegPNWfZlF6w+D7gz8O8d9FGSJEkaOyMv3qvqiCRPBj4GvAuY7DZaAS4AnlZVR4yyf5IkSdK46mSe96r6ZpLvADsDDwQ24ZabNJ0PnAz8uKpu6qJ/kiRJ0jjq7A6rbWF+TLtIkiRJWo4uLlj9F0luk2SdJJ19mJAkSZLGXWfFe5KnJflekotoLla9DLghyUVJvp/kP7rqmyRJkjSORn6mu50C8jvAw2jGuP8KOB64HlidZraZnYFHJnke8NiqunbU/ZQkSZLGTRfDVN4B7AS8DPh0Vd0wcYMkqwH7Ah8A3g68dqQ9lCRJksZQF8NmngJ8tKo+PlnhDlBVN1TVx2imk3zaSHsnSZIkjakuivcNgd/PcNvTgQ2G2BdJkiRpzuiieP8z8MgZbvvodntJkiRpweuieD8EeGKSryXZIckqvY1JVknykCRfBx7fbi9JkiSNnSSzWmariwtWPwxsBrwEeCJwc5JLaKaLXI1mmMyyDxWfaLeXJEmSFryRF+9VVcDLk3wKeDqwCNgEWAO4Evg1cDLw1ao6bdT9kyRJksZVZ3c0rarTgbd0lS9JkiTNNZ3dYVWSJElSfzor3pPcOclrknw1yYlJftl+/Wq7ftNZ7nenJN9Jcl6SSvLsCe1JckCS85Ncl+T4JNsM5EVJkiRJQ9RJ8Z7krcBZwH8BTwK2BNZvvz6pXX9WkgNmsfu1gNOAlwPXTdL+OuDVwEuBBwIXA0clWXsWWZIkSdLIjLx4T/JC4ADge8BOwG2raqOquktVbQTcFti5bd8/yYv62X9VHVFVb6qqbwA3T8gO8ArgoKr6f+0Fsc8C1qa5eFaSJEkaW12ceX8J8M2qelJVnVhVN/Y2VtWNVfWTqnoi8G3gxQPM3hzYGDiyJ+864ARghwHmSJIkSQPXRfF+V+AHM9z2++32g7Jx+/WiCesv6mmTJEmSxlIXxfvfga1muO3W7fadSbJvksVJFi9ZsqTLrkiSJGmB66J4/xbwsiT7JVl1sg2SrJLkxTQXlX5zgNkXtl83mrB+o562W6mqQ6pqUVUt2nDDDQfYFUmSJKk/Xdyk6c3Ag4GPA+9NcipwHnADsBpwR+D+wJrAKQz2Rk7n0BTpe9DcxZUkqwMPBV47wBxJkiRp4EZevFfVlUkeAjyfZoaXbYEdeza5Dvgl8GXg01X1j372n2Qt4G7tw5WATZPcD7i0qs5N8mHgTUnOAP5A8+HgauBLs35RkiRJ0gh0ceadtiA/GDi4nb7x9sAawLXAZVVVK7D7RcBxPY/f3i6fB55NM4f8bYFPtLk/Bx5eVVetQKYkSZI0dJ0U773aQv3SdhnE/o4Hspy8A9pFkiRJmjM6ucOqJEmSpP51Urwn2SrJ15KcnuS4JM+aYrtnJFk66v5JkiRJ42jkw2aS3Ak4CbgdzVCZuwI7JXky8LSqunrUfZIkSZLmgi7OvO8PrArsUVUbABsCHwAeBRyd5HYd9EmSJEkae1OeeU9y7Cz2V1W123K22Rn4ZFUd0z7hSuB1SX4BfJGmgN+jqi6fRb4kSZI0b003bGYLYOKUjWsCG7TfX95+Xbf9egnNfOnLsylw2sSVVfWNJNcC/w84JsnuM9iXJEmStGBMOWymqjarqs2XLcBuNDdQ+giwSVWtV1XrAZsAH6WZo315Z92hKfI3mKyhqo4A9gLuCRwDrN/Pi5EkSZLms37GvH8I+L+qemVVXbhsZVVdWFWvAH7WbrM8vwceNlVjVf2QpoDfCjioj/5JkiRJ81o/xfsuwI+naT++3WZ5jgB2T7L5VBu0BfzjmeZmS5IkSdJC089UkQVsPU37NjPcz+eBM4Bp52+vqiOT7Ajca4b7lSRJkua1for3I4EXJTkF+EJVFUCSAHsDLwAOX95O2llkfjSTwKo6BTiljz5KkiRJ81Y/xfurgAcCnwMOSnJWu35LYCPgr+02kiRJkoZgxmPeq+pvwP2A9wKXAQ9ql8vadfdrt5EkSZI0BP2ceaeqrgDe1C6SJEmSRqif2WYkSZIkdaiv4j3JnZN8NsnfktyY5GHt+g3b9Q8cTjclSZIkzbh4b+dlXww8EfgdsPKytqpaAiwCnjfoDkqSJElq9DPm/d3AzTTzrl8HXDyh/QjgsQPqlyRJkqQJ+inedwc+VlV/TbL+JO1/Ae403Q6SnN1P51pVVXedxfMkSZKkeaWf4n0d4IJp2ledwf7OpblTa687AXcFrgSWFfdbtHl/Apx+UpIkSaK/4v2vwDbTtG8H/HG6HVTVLr2PkzwAOBp4BfDJqrqxXb8qsB+wP/DUPvooSZIkzVv9zDbzTeA5Se7Vs64AkjwReDLwtT7z3w98rao+uqxwB6iqG6vqw8A3gPf1uU9JkiRpXuqneH83zRCWnwNfpCnc35DkJJqi/dfAB/rMfxDwq2naT223kSRJkha8GRfvVXUlsD3wPzTTQgbYA7gHcDCwa1Vd32f+dcCDp2nfHuh3n5IkSdK81M+Y92UF/MuBlyfZkKaAX1JVEy9CnanDaYbinAN8sKquBkiyFvBq4JnAZ2e5b0mSJGle6at479XemGlFvRa4L3AA8JYky2azuQNN337ZbiNJkiQteH0X70m2BLYE1qc5834rVXXYTPdVVZcn2QF4DvA4mikiAY4Cvg18rqr+0W8fJUmSpPloxsV7ko2Az9OMc4dJCneai1hnXLwDVNVNwCHtIkmSJGkK/Zx5/zhN4f7fwLHA3wfdmSQbAFTVJYPetyRJkjTX9VO870FzI6WXDLIDSTYBDqQZNrN2u+5KmmEzb66q8waZJ0mSJM1V/RTvK9HM5T4wSTYFfgZsTDPf++/apnsCewN7JNmuqv46yFxJkiRpLuqneP8Jzcwwg/RO4PbAnlV1RG9DkkfR3NX1ncCzB5wrSZIkzTn93GH1VcBeSZ44wPyHAwdPLNwBquoHNOPrHznAPEmSJGnOmvLMe5JjJ1l9NfC1JOcDZwNLJ7RXVe3WR/7tgbOmaT8LWLeP/UmSJEnz1nTDZragmfpxonPbr5sOIP9vwC7AJ6do36ndRpIkSVrwpizeq2qzEeR/HXhdknOAg6rqCoAk6wBvAJ4CHDSCfkiSJEljr+87rA7YO4GHAq8HXtMOxwHYBFgZ+Cnwro76JkmSJI2Vfu6wujKwWlVd27NuXeC5wHrAV6rqt/2EV9W1SXYB9gEeD2zeNv0IOBw4tL0DqyRJkrTg9XPm/VPAdsC9AJKsApxIMyc7wKuSbF9Vv+qnA21x/ul2kSRJkjSFfqaK3BH4Ts/jJ9EU7i8GdgAuohmnPtaSrJzknUnOSXJ9+/VdSboeQiRJkiRNq5+C9Q7AOT2PHwP8rqr+GyDJIcALpttBkr3bb79QVdXzeFpVdVgf/Vye19N84HgW8FvgPsDngRtoxuBLkiRJY6mf4j00F5EuswvNHVCXuQD4t+Xs41Ca6Se/AtzY8zjTPKeAQRbvOwDfrarvto//nOQ7wIMHmCFJkjQWkunKrOlVTTZruLrUT/F+DvAI4JNJHkJzJv64nvZNgCuWs49dAarqxt7HI3YisF+SrarqjCT3BB4GHNhBXyRJkqQZ66d4/xzwwSSnAXcELqaZFWaZBwNnTLeDqvrxdI9H5L3A2sDpSZbS/AzeXVUHT7Zxkn2BfQE23XQQ96WSJEmSZqefC1Y/AryNZmz4qcBey6aNTLI+zUw0R8x0Z0nWSvKnJK/oow+D8FRgb+DpwAPa7/dL8tzJNq6qQ6pqUVUt2nDDDUfYTUmSJOnWZnzmvZpBT+9kkos6q+rvLH+8+8TnXN0W/Vf387wBeB/w/qr6Svv4t0nuArwR+MyI+yJJkiTNWD9n3ofhZ8CiEWeuASydsG4p3f8sJEmSpGlNeeZ9RNM6vgE4NsnPae6mOopLmr8LvCHJOcDvgPsDr2KwM9pIkiRJA5ep6uUkN9NM03jbqrqx5/G00zpW1crTtE/MOBa4C7AZcCnwJ+DaSfa520z3OYPMtWmG/uxFM9TnApqpK99RVddP99xFixbV4sWLB9UVSZKkoetiqsiFkrkiudNlJjmlqiYdnTLdmPdRTOu4Bc0HgnPbxxsNIeNWquoq4BXtIkmSJM0ZUxbvo5jWsao2G/Q+JUmSpPnKizQlSZKkOWK6C1Z3ms0Oq+qEfp+TZDNgd5phM/9bVX9OsiqwMXBhz9AdSZIkacGabsz78TTj0Wcq7fYzvmAVIMl7aWZ7Wbl9/knAn4HVgdOBtwAf7mefkiRJ0nw0XfG+z7DDk7wAeC3wUeB7wJHL2qrqyiTfAR6LxbskSZI07QWrnx9B/n7At6rqFe3dVif6DfCSEfRDkiRJGntdX7B6d+CoadqXABuMqC+SJEnSWJtu2MykkjyF5gZHW7SrzqY5e/61WeRfD6w5TftdgMtnsV9JkiRp3pnxmfckayY5Cvgy8FRgy3Z5KvDlJMckma4Qn8wvaD4ITJa3OvCfwE/73KckSZI0L/UzbObdwG7Ax4BNqmq9qloP2KRdt2u7TT/eB2yf5AvAfdp1Gyd5BM1sN3cC3t/nPiVJkqR5KVUzmw0yyQXACVX11Cnavw7sWFV36KsDyb7AR4BVuWW6SYAbgRdV1aH97G+YFi1aVIsXL+66G5IkSTOWZNbPnWmduFAzVyR3uswkp1TVosna+hnzvg5w3DTtxwKP7mN/AFTVIe2UkE8GtqIp4M8CvlZV5/W7P0mSJGm+6qd4/w3NGPepbAn8djadqKoLaYbeSJIkSZpCP8X7W4BvJTm+qr7b25DkccDzgMfPtiNJ7kHPDDZVdeZs9yVJkiTNR1MW70k+O8nqc4DDk5wJ/L5dtzVwD5qz7s+gGT4zY0keRnPWfasJ688AXlZVx/SzP0mSJGm+mu7M+7OnaduKCcU2zWwx9waeO9PwtnD/IXAD8Gng9LZpG+A/gB8keWRV9fWBQJIkSZqPpizeq2oUd199D3ARsN3Ei1OTvBP4Gc30k9uPoC+SJEnSWBtFgT6d+wCfmmxWmar6G/Ap4L4j75UkSZI0hrou3q8Arpqm/Urg8tF0RZIkSRpvM55tJslMxp1XVe3WR/7Xgf9I8omqumlC3io0496/3sf+JEmSpHmrn6kit+CWu5/2Pv8ONGfwLwGu6TP/k8AOwAlJPgSc0a7fGnglsDLwySSb9j6pqs7tM0eSJEma82ZcvFfVZpOtT7Ia8CpgH2DnPvNPo/lAEOArE3fds81EK/eZI0mSJM15/Zx5n1RV3QAcmOSewAdphrrM1Dv417P5kiRJkiaxwsV7jxOBA/t5QlUdMMB8SZIkaV4b5GwzmwOrDnB/kiRJknr0M9vMplM0rQfsDrwMOH4AfZIkSZI0iX6GzfyZqcenBziTpoCXJEmSNAT9FO+TXVxawKXAH4Cjq+rmQXVMkiRJ0q31M1XkAUPshyRJkqTlGOQFq5IkSZKGqK/iPcnqSV6X5KQkF7XLSe262w6rk5IkSZL6m21mQ+BYYBvgSuDstmlr4MHA3kl2raol/XQgyZrA04EtgfW55c6qy1RVPbeffUqSJEnzUT8XrL4PuCfwKuDgqroRIMmqwIuB97fbPHumO0zyIOB7wAbTbFaAxbskSfNYMvHc3cxUeaN2LSz9FO+PBT5TVR/uXdkW8R9Ksg2wV5/5H6S5sdNTgGOr6tI+ny9JkjRnzPZDCvhBRY1+xryvCvxymvbF9H+H1W2BD1TVNyzcJUmSpOn1U7yfDDxgmvZtgV/0mX8l8Pc+nyNJkiQtSP0U768GnpTkpUn+OdwmyW2SvBx4QrtNP74JPKLP50iSJEkLUmY6firJscCdgS249WwzWwDrAH8C/jbhaVVVu02zz3WAH9EMufkwcHaN8YCuRYsW1eLFi7vuhiRJ885CuWC1izHvZg4vc0Vyp8tMckpVLZqsrZ8LVregmfnl3Pbxeu3Xy9tlFWDzPva37LkFPAjYr+3sxG2qqvrppyRJkjQvzbgorqrNhpB/GE3xPlJJ7gAcBDwaWJvmrwgvqqofj7ovkiRJ0kx1eka7qp496swk6wI/BU4EHgMsofmrwsWj7oskSZLUj4U4HOV1wAVVtXfPunO66owkSZI0U/3MNjNfPB74eZKvJrk4ya+SvCQrcpWDJEmSNAIjLd6T3JzkpiSr9jxeupzlpgF3Ywuai2PPppmm8iM0499fPEWf902yOMniJUuWDLgrkiRJ0syNetjMsgtUl054PEorAYur6o3t41OTbElTvH984sZVdQhwCDRTRY6sl5Ikaai6mlpQWhEjLd4nXqDaxQWrwAXA6RPW/R54eQd9kSRJkmZsIY55/ylwjwnr7g78pYO+SJIkSTM24+I9yd2SPHLCugcn+W6SnybZd/DdG4oPAdsleXP7mp4MvAz4RMf9kiRpLCSZ1SJp+PoZNvNemruq/hAgyQbAD4C1gOuA/05ycVUdPuhODlJVnZzk8cB7gP1p7hi7P3Bwl/2SJEmSlqef4n0R7YWbrf8A1gHuB/wBOJ5m3Pjhg+na8FTV94Hvd90PSZIkqR/9jHnfEDi/5/EjgZ9W1WlVdSPwFeCeg+ycJEmSpFv0U7xfA6wLkGRlYEfghJ7262jOxEuSJEkagn6K998BeydZH3g+zVj3o3ra7wJ4FyNJkiRpSPoZ8/4+4NvAxe3jU4Gf9LQ/HPjldDtIsmlfvWtV1bmzeZ4kSZI0n8y4eK+q7yd5GPA44Arg49XeXqw9G/83mjumTufPzO6OqivP4jmSJEnSvDKj4j3JasCDgQuq6tUT26vq78ATZrCrdzC74l2SJEla8GZ65n0pcAzwauCs2YZV1QGzfa4kSZK00M3ogtWqugm4EPD2aZIkSVJH+plt5uvAU5L08xxJkiRJA9LPbDP/A+wKHJXkwzTDZ66duNF0M8MkuZn+x7xXVfXTT0mSJGle6qcoPo2m8A6wyzTbTTczzGF4waokSZI0K/0U7ys8U0xVPXtFni9JkiQtZP3M837AEPshSZIkaTm8+FSSJEmaI/oq3pOsneStSU5MclaS7dv1G7Trt+q3A0kekuR7SZYkuSnJ0gnLTf3uU5IkSZqPZjxsJsmGwInAFsAf26+3BaiqS5I8C1gXeFUf+9wJOBq4Avg58GjgWGAt4EHAb4FfznR/kiRJ0nzWzwWr7wI2Bh4MnAtcPKH928Bufea/GbgAWERzMezFwHuq6tgkDwe+AezX5z4lSZKkeamfYTN7AgdX1S+ZfNaZs4E795n/IOB/qmoJcHNvn6rqSOALwDv73KckSZI0L/VTvG9AM1xmKjcDq/eZvxpwXvv9De3XtXvafwVs2+c+JUmSpHmpn+L9QuCu07Tfn2Y4TT8uAO4EUFXXAJcD9+ppvxPgBauSJEkS/Y15PwJ4bpKPATf2NiR5MLA38OE+808GHtLz+EjglUn+QvPB4iU0F7JKkiRJC14/Z97fTnMW/FTgQJpx789K8mXgBOB84L195n8GuCTJbdvHbwKuAw4FPkszlOZ1fe5TkiRJmpf6ucPqhUm2Az4OPAcI8J80RfwRwIuq6tJ+wqvqKOConsdnJ7k7zaw1S4ETq+qKfvYpSZIkzVf9DJuhqv4KPC7JOsA9aAr4P/ZbtC8n4xrgO4PanyRJkjRf9FW8L1NVV9KMV5ckSUOUZFbPq5psVmdJc13fxXuSBwF70dxhFZr53Q+vKi8slSRJkoZoxsV7kpWBQ4Bn0wyX6fW6JIcBz6uqpYPrniRJkqRl+plt5i3APsC3gR2AddvlITRj1Pdut5EkSZI0BP0U788BjqqqJ1TVz6rqynY5qar2Ao5tt5EkSZI0BP0U7//G9LPAHN5uI0mSJGkI+rlg9Q/AxtO036Hdpm9J1gA2A9bnX8fTU1UnzGa/kiRJ0nzST/F+IHBwkm9U1a97G5LcH9gPeFE/4UnWBD5EM15+lck2obkJ1Mr97FeSJEmaj/op3u8OnAMsTnIkcEa7fmtgD+DXwD2SvLXnOVVV75xmn58EngF8C/gJcFkf/ZEkSZIWlMz0Jg5Jbp7F/quqpjxrnuRK4KtV9fxZ7HvkFi1aVIsXL+66G5KkBaSLmzSZObxcM+dX5orkTpeZ5JSqWjRZWz9n3jfvt1Mz8A+8U6skSZI0IzMu3qvqL0PIPxZ4MM3NnyRJkiRNo5+pIofh1cBuSV6eZLILViVJkiS1+hk2Q5LbAI+nOVt+e/61+K+qeu5M91dV5yZ5E3AY8L4kFwBLJ9nnXfvppyRJkjQfzbh4T7IecBxwL26ZwnHZCP3qWTfj4j3Js4HPADcCZ9LBbDNJ3gi8B/hEVb1k1PmSJEnSTPVz5v1dwFbA84DjgT8BjwDOBfYHtmwf9+PNwK+AR1TVJX0+d4Ul2Q7YF/jNqLMlSZKkfvUz5v0xwGFV9Tngynbd0qo6s6qeCVxHcyOnftwR+ExHhfvtgP8FnoPzy0uSJGkO6Kd435hbpnW8qf26ek/74cC/95l/JrBen88ZlEOAb1TVcR3lS5IkSX3pp3i/FFiz/f4qmjna79zT/g+ai1j78R5gvyR36vN5KyTJ84G7AW+Zwbb7JlmcZPGSJUuG3zlJkiRpCv2Mef8DcE+Aqro5yanAs5McCqwM7A2c3Wf+1sB5wO+TfAs4h8lnm3lnn/udUpJ70Hxo2LGq/rG87avqENp56BctWjT7229JkgamqzspSlLX+inejwRek+QlVXUD8EHgKzRn5Au4Lc3Fn/04oOf7Z06xTQEDK96B7YENgN/1HPxXBnZK8kJgzfb1SZIkSWOln+L9PcD7lxW2VfW1JDfRFN1LacaPf7XP/M373H4QDgcWT1j3OeAsmtd446g7JEmSJM3EjIr3JBsCWwCX0EwRCUBVfRP45mzDq+ovs33uCmReDlzeuy7JNcClVXXaqPsjSZIkzdS0F6wmWSnJJ4ELgP8D/pDkxLaYH6gk6ydZ1C7rD3r/kiRJ0ly3vDPvL6EZx34+cBLNjZh2AD4FPGEQHUhyX+CjwI4T1v8EeFlVDf0GSlW1y7AzJEmSpBW1vOJ9b+D3wHZVdRVAkk/TzDKzbjsEZdaS3As4kWa++G8Dv2ubtgEeC/wkyQ5V9bspdiFJkiQtGMsr3u8BvGNZ4d76GPBc4O7AL1Yw/x0088M/ZOIZ9rawP6Hd5okrmCNJGhKnbZSk0VneTZrWpBky0+v8nrYVtRPwicmGxrQXjx4M7DyAHEmSJGnOm8kdVieeFln2ePanWm6xJnDhNO0XMJgPCZIkSdKcN5OpIh+dZOOex2vQFPBPTnK/CdtWVX2oj/yzgT2BT0zRvif937VVkiRJmpdmUrw/vV0mesEk6wrop3g/DDgwyZeAdwNntOu3Bt4IPBx4Qx/7kyRJkuat5RXvuw45//3AA4CnAU8Fbm7Xr0QzLOdrwAeG3AdJmje8eFSS5rdpi/eq+vEww6tqKfDUJP8DPB7YvG06Gzi8qo4eZr4kSZI0l8xk2MxQJFkTeDXw86r6EXBUV32RJEmS5oKZzDYzFFV1DfAm4M5d9UGSJEmaSzo78976E7DxcreSJKljXk8gaRx0dua9dTDw/CTrd9wPSZIkaex1feb9KuBS4MwknwfOAq6duFFVHTbqjkmSJEnjpuvi/dCe7185xTZFMx+8JEmStKB1XbwPex55SZIkad4YafGe5K3AN6vqtHbVOcCSqrpulP2QJEmS5qJRX7B6AHCfnsfnAHuNuA+SJEnSnDTq4v1yYN2ex7Ofd0uSJElaYEY95v1U4HVJVgEua9c9NMm0/XC2GUmSJGn0xfurgG8CH2ofF/CCdpmKs81IkiRJjLh4r6pfJ7k7sAVwB+B44N3A0aPshyRJkjQXjXyqyKpaSnMzprOS/Bg4vqp+POp+SJIkSXNNp/O8V5XzvEuSJEkz1PVNmiRp3kpmP6FWVQ2wJ5Kk+WLUU0VKkiRJmiWLd0mSJGmOsHiXJEmS5giLd0mSJGmO6LR4T3Jskt2mad81ybGj7JMkSZI0rro+874LsNE07f8G7DyarkiSJEnjrevifXnWBW7ouhOSJEnSOBj5PO9J7gPcr2fVQ5NM1o/1gP2A00fRL0mSJGncdXGTpr2At7XfF/CCdpnMVcDLRtEpSZIkadx1UbwfChwPBDgWeA9w1IRtCrgaOL2qrh9l5yRJkqRxNfLivar+AvwFIMk+wI+r6s+j7oekhSXJrJ5XVQPuiSRJs9fFmfd/qqrPd5kvSZIkzSWdFu8ASdYEng5sCaxPM5ymV1XVc0feMUmSJGnMdFq8J3kQ8D1gg2k2K8DiXZIkSQte1/O8fxBYFXgKsEFVrTTJsnLHfZQkSZLGQtfF+7bAB6rqG1V16SgCk7wxyclJrkyyJMl3k9xrFNmSJEnSiui6eL8S+PuIM3cBDgZ2AB4G3AQcnWS9EfdDkiRJ6kvXF6x+E3gETTE9ElX1iN7HSf4TuAJ4CPDdUfVDkiRJ6lfXZ95fD/xbko8luWtmOxHzilmb5udwWQfZkiRJ0ox1feb9cprZZB4E7AeT3kilqmqY/fwI8CvgpMkak+wL7Auw6aabDrEbkiRJ0vS6Lt4PoyneO5Hkg8COwI5VtXSybarqEOAQgEWLFnmrRUmSJHWm6zusPrur7CQfAp4G7FpVZ3fVD0mSJGmmuj7z3okkHwGeSlO4n9F1fyRJkqSZ6PqCVZKsnGTvJF9MclSS+7frb9+uv+OA8z4B7AM8HbgsycbtstYgcyRJkqRB67R4T7IG8GPgUOBxNPOu375tvhI4CHjRgGP3o5lh5hjggp7lNQPOkSRJkgaq6zPvBwCLgL2ALYB/TjXTXkC6bB74gamqTLEcMMgcSZIkadC6Lt6fDBxSVd8Gbp6k/Y/AZiPtkSRJkjSmui7eNwF+PU37tTRDXCRJkqQFr+vi/e/AdBekbgOcP6K+SJIkSWOt6+L9GGCf9sLVW0myOfAc4Icj75UWvCSzXiRJkoal63ne3w4sBk4Gvkxzt9VHJtkDeCFwA3Bgd92T5r/ZfuCo8obDkiSNWqdn3qvqj8BuwE3AO2hmm3kN8Hrgr8BuVfXX7nooSZIkjY+uz7xTVacA901yL2BrmgL+rKo6tdueSRoWz/ZLkjQ7nRfvy1TVacBpXfdDkiRJGldd32F1tyRTjmlPcmCSXUfZJ0mSJGlcdT3bzOuBu03Tvnm7jbQgOMONJEmaTtfF+32Bn03T/vN2G0mSJGnB67p4vx1wzTTt1wG3H1FfJEmSpLHWdfF+HrDtNO3bAheOqC+SJEnSWOu6eP8+8Kwku09sSLIb8CzgiJH3SpIkSRpDXU8V+W7gicCPkvwA+FW7/n7Ao2jOur+zk55pbKzIBZnOCy5JkuaTTov3qrooyUOAg2mK9UcvawJ+ALykqi7oqn+SJEnSOOn6zDtV9Wfg0Uluzy3TRv6xqi7rrleSJEnS+OlszHuStZIsTbI/QFVdVlUnt4uFuyRJkjRBZ8V7VV0NXA5c3FUfJEmSpLmk69lmjgN27rgPkiRJ0pzQdfH+WmDHJG9Psk7HfZEkSZLGWtcXrB4DrA68BXhLkiXAtRO2qaq668h7JkmSJI2Zrov3c2mmhZQkSZK0HF3P875Ll/mSJEnSXNL1mHdJkiRJMzQWxXuSnZK8K8mnk2zVrlurXb9ux92TJEmSxkKnw2aSrAx8CXgSEJrx718GzgBuAg4H3g+8p6MujrUks35ulZcaSJIkzTVdn3l/PfBE4FXA1jQFPABVdT3wLeDR3XRNk0ky60WSJEkrpuvifW/gsKr6CHDJJO2/B5wmUpIkSaL74n0z4KRp2i8Hbj+SnkiSJEljruvi/SpgvWna7wYsGVFfJEmSpLHWdfF+IvDMTDIgOsntgecAx428V7PgWHBJkiQNW9fF+7uBLYFjgT3bdfdN8gLgl8CawEEd9U2SJEkaK13fYXVxkicC/wN8rl39fppZZy4G9qqq07vqnyRJkjROOi3eAarq+0k2Ax4ObEVTuJ8F/Kiqru2yb5IkSdI46bx4B6iqG4DvtoskSZKkSYxF8Z5kNWAXYIt21dnAj9sbNUmSJEliDIr3JHsDH6SZz33Z1CsFXJ7k1VV1aFd9kyRJksZJp7PNJHkqcChwNfBm4PHt8pZ23WfabYaRvV+Sc5Jcn+SUJA8dRo4kSZI0KF2feX8TcAawXVVd2bP+O0kOBn7ebvPVQYa2Hwg+AuxHM9f8fsAPktyzqs4dZJYkSZI0KF3P834P4HMTCncAquoKmukj7z6E3FcBh1bVp6vq91X1UuAC4EVDyJIkSZIGouvi/cLltBdw0SADk6wKbAscOaHpSGCHQWZJkiRJg9T1sJlDgX2S/HdVXd3bkGQdYB9uuXnToGwArMy/fii4CNh94sZJ9gX2bR9eneTMWWZeMlVjkqmaVkQXmdPmmmmmmTPLHGKuxwUzzRxA5hBzF0rmtLlmAnCXqRq6Lt5/AuwJ/LYd435Gu35rmiEslwA/SbJT75Oq6oRRdbCqDgEOWZF9JFlcVYsG1KWxzewq10wzzRzfzK5yzTTTzPHN7Cp3vmR2Xbwf1fP9e2mGycAtU0beZcI2abdZeQUyLwGWAhtNWL8Ryx/GI0mSJHWm6+J9n1EHVtWNSU4B9gC+3tO0B/D/Rt0fSZIkaaY6Ld6r6vMdRX8Q+EKSXwA/BV4IbAJ8ckh5KzTsZg5ldpVrpplmjm9mV7lmmmnm+GZ2lTsvMlNVy99qHkqyH/A64A7AacArRzmWXpIkSerXSIv3JLtV1TGzfO7uVXX0oPskSZIkzRWjnuf9h0mOTbJnkuVedJpklSR7JfkxcMQI+idJkiSNrVGPeb8/zXjz7wBLkhwN/AL4E3ApzWwy6wFbAtsBuwHr0txA6X4j7qskSZI0VjoZ855ke2A/4HHAWtwyReQ/NwGuBL4J/HdVnTzaHkqSJEnjp9MLVtuhM9sC9wQ2pCnil9BcQHpqVd3cWedWUJK7ABu3Dy+sqr902Z/5LMlKwMpV9Y8RZj4P+HpVXTGqzFFL8kBgU+AvVbV4CPvfoKqmvKvfMCVZr6ou7Xm8A80wwlOq6roh5m5Kc5H8zcDZVfX3YWW1eWvRHGP/eSyieY1XT/2sofbnNsAmVXVuF/nDluSOwGpVdfYIM98JfLSqlowwc0Pg8lEcc5OsDjyW9lgEfK+qrh9wxrZVdcog9znD3DWBewC/q6obktwW2IvmWHRsVZ0/xNxt6TkWAb+sIReE43Q8mvPHoqpyGeACvBL4K82NoG5ul6Xtuld00J/7AkuHtO+n0cyVfwiw7YS2DWiKk0HmrQIcRDO957vbdW8ArgNuBP6X5o1zFD/XG4Gth7TvhwO36Xn8dOBXwDXAH4GXDSHzPcAj2+83BE5qf3f/0X79KbDhgDOXAkcDTwVWGdG/25bAH9rsk9rf0+N6/q/+GdhqCLn70RQeSycsJ078vzOgvNsAHwGu7fl3XPZveS3w4VH9zCf0a+DHI5q/1L4BOJlmiOXjJ7RvNITMdYCvAOctO+7QTDW87Hh/InC7AWeuN8myfnss2n7ZugFn7rvsmNr+nN8EXNa+xmtohsGuNODMQ4G92u83b/9PXtse+66hKTQ3H3DmzcBZwOuBfxvkvqfJfCDNcOFlxfPdgNOBq4Gr2p/zgwecuRLwX+3PcdkxqPfY99ghvdaxOx7N9WPRyH5QC2EB9qcZ7vNmYBFw53ZZ1B70rgDeMuI+3Re4eQj73af9z/dV4AfADcC+Pe3DeMN8N80n9Q8Av2vfLP8KPAPYG/gb8PoBZ145xXJze5C9ErhywJlLl72BAE8EbgI+0b7GDwHXA/8x4MzzgPu23x8K/AzYsn18V+B44LABZ95MUzjfBFwMvB+4x6B/VydkHg78CNgF+Fx7kD0WuCPNWagfAt8ecOZr2p/vS4Dn0bxB7w88EjiM5o100YAzP9JmPgPYoGf9BjQfBv8GfGiYP+sp+jWMN8zXtv8P3wv8N82H+Xf3tG806GMg8HHg98BL29/hbwO/BR4C7NQen94z4MyJH/x6i69/fh1C5rJj0QvaY95r29f4cuBy4CUDzlwCbNN+/03gu8A67eO1aE4YDfr/6M3t/8XLaN7L/h/wiEFmTJJ5NPAFYDOakydnAV+jOUl1m7btmAFnHtQef/YEdgd+TDNl9lbAO2jeWx4+hNc6dsejuX4sGtkPaiEsNIXkE6ZpfwJw3oAzj13O8otB/4K2ub8GXtjzeE+aDycvbR8Po3j/E7Bn+/3d2jeWp/a0PwX47YAzr2rfPJ7VszybpuB807J1A868uecN80Tg7RPaXwP8YsCZ1wObtd+fAzxkQvv9aYZ/Dfx10nzAPYBbzkz/BPhPYPVB5rWZS4D7tN+v3ebt2NP+AOCiAWeeAzyq5/HdgUto/7rSvrEdOYTXuds07bsDS4bw8z17OcvfhnBcOAN4yoR/w/OB97ePh3EsOhfYtf1+k/Z3ec+e9scAZww487z2WLQrsHO77NIei569bN2AM3uPRb+guR9Kb/vzgF8POPM6YIv2+78x4YMtzTDbvw/jdQK3bX+WP+WWs9H7A3caZF6beRntyQqav9zcBDyop32bIbzO84GH9jy+I8173LK/ruwP/N8QXuvIj0fz/VjU6R1W56H1ac7GTOVM4PYDztwR+B7NgX0yG9CMMRu0u9GcpQSgqr6X5NHAEe0Y9K8MIXMTmg8NVNUfk9y47HHrZOAuA858APBlmjNNL62qawGSfBo4vKpOH3DeRFsCL5uw7jvAWwaccw5wL5o3q8muNSmaN7aBq6q/AgckeQfwKOD5wGeBjyb5YlW9dIBxq9GcPYTmjPfNNG9ey1zJ4F/nv3Hr48JZwO1ohiddQPNaTxxw5m1pPiBM5RKG8+95B5ozmH+Yov2ONGdsB2lTmsISgKr6ZZJdgePa66oOGnAeNP+mf2zzzk9yHbd+zafRfCgdpPvQ/LXoTcB/VtWFAEmK5sP8sI5F1X7dHJh4n5Zjaf4aOEhnAA+mKbCu4F/fM9flXye5GIhqrnc5FDg0yTY0w4ZeAbw1yQ+r6rGDjpzw9aaetqU0wzAGaS1uXStcCKxO8zO+kOYvDm8YcCZ0czya38eiQX/CWsgLzdCCLwOrTtK2KvAl4PgBZ/4aeN407fdjOGfezwe2n2T9jtzyZ6NBf6q9ELh3z+OfAnfseXwP4IohvNZVaN6gzqQ9C0QzZOieQ/o9upnmTMQDaIrpB01o3wq4asCZr6B507x7+/1JwF3bts1phgZ8dcCZ//yT/CRtd6D5gDLo6yb+DziQ5k3x+TTF84E97fsDJw8481TgBT2P96D54LBswoCtGPzQq+/SFFabTNK2Cc2f7L8zyMx23ycDL56mfRh/qj4X2GmS9Vu1/75fGkLmecD9ex5/qfd3meas6aWD/vm2+345zVnDx7ePh30s2ofmr8Z/BXaY0L7NoI+5wDPbn+/DaIZZnE4zbfQmNH91+A3wqQFnTncsWq3t0/EDzjyK5sPYXWiGrJwJHNrTfjBwwoAzfwq8tefxM4DLeh7faxi/t10cj+b7scgz74P1Epr/kBcn+QlwUbt+I+ChNBdm7DHgzF/SFHlTuYHmF2rQfkFzlvSk3pVVdWKSPRnOTbVOpxm+8ds26yET2u9NezZskKqZUeGV7X0Jvp3kYIZ05qfHj7jlrMtD6Pk0T/MzGOi/aVV9uJ0J5TSa4UmbAX9IchPN+Mtf0lygPEhTnlWqqguAd7XLIL2dZtz7a2iKnocDn0nycJo38AfQjMEcpAOB/20zrqeZIvej1R7VaYY+nDbgzP1o/g+em+T33PpYtDXNmOzHDDgTmuLg7tO0Xw2cMODME2mKy1vtt6rOSLIbzQfPQfstzQWHp7ZZE39ntqX5MDxwVfWRJCcAX04yjH/DiT7T8/2uNB+Al9me5ngxMFX1xSTr0fyFcSVgZZqL/5b5DvCqQWYy/bHoBuCL7TJIb6K5XmxvmmEluwKfTXIRzfvLOjSz7AzSW4HvJ3kczbHowTTHwmUeSfs7PWBdHI/m9bGo06ki56Mka9N8St+OW0+HdBLwpaq6csB5q9FMk3jtIPc7g9ydac7CHDhF+y40Y8H3GWDm1sCNVTXpm0WSZwI3VdUwhuwsy9iE5kKiXYF71RD+VN1OM9rr6uqZUjDJ3gBVddgQsu8B/DuwBc0b5wU0B8Gja8AHiyRvA97Xwe/uZjTF1clVdW6SjYAXA2sA36+qgRd7SR5Fc1xYDfhRVX26p219gBrwtJHt8LVHMPmx6Miaw1Px9kpyH5oZez43Rfs2wJOq6u0DzNyA5sKzS6dofwxwfVVNHGYyMEnWAD5Gc4Z6j6oa+ImLGfRhT+AfVfWjIez7djQfrm91LKqqs4aQ9SzgK22hPjLtlI1bAWdW1dXttJjPoBlCclRVnTmEzPvSXB+27Fh01KAzpsid98ejUR6LLN4lSZKkOcJhM0OQZEtgB2796fL/hnHGoMvMrnLHJPMC4KQF8DoXUuZPh3n2sqv/o1P0ZU2aM0SD/rPx2OWaaaaZ45vZ5q5EM6PPyG6YNOczBzlYf6EvNDNIfJtbZrD4U7tcRTOe9nDa+WrncuZCeq1mmjkXM2fQp4FfrDWuuWaaaWa3mTQz2nySZmz/H4CXT2gfxnSu8zpzJTRIH6O5oc1Dq2rtqrpru6xNc8HqFu02cz2zq1wzzTRTkuaWt9LcC+atNFPjviXJF9sz0csMelrM+Z056E9YC3mhudvclLczprky//K5nrmQXquZZs7RzKnuxnmrZZCZXeWaaaaZ45vZ5v6JW9+o7i40s8d9heZi5GHd1HHeZjrmffCmuwJ4WFcHd5HZVa6ZZpq5fNcBH2bqad82p7kXw3zINdNMM8c3E5r7dvzzRnVV9Zf25kXH0hS2g576c/5nDvoT1kJeaKYQ/C2w3SRt29HcUOmwuZ65kF6rmWbO0cwTmXAr+wntwxrbOvJcM800c3wz2/3+Cdh9kvUb0xS7xw7htc7rTMe8D9ZLae58939Jrkjyx3a5gmau7PPabeZ6Zle5Zppp5swcQXOh7FQupbl1+KB1kWummWaObyY0Nyf6l5vfVdWFNPcpuJOZ/XGe9yFobya0Pc34JmhvRFBVQ7njXleZXeWaaaaZkjQ3tDce3KqmuJlXkjsAD6+qz5s5wyyLd0mSJGlu8ILVAWtvcvB0JrkBDPDlqrpmPmR2lWummWaOb2ZXuWaaaeb4ZnaVO58zPfM+QEnuCRwFrA2cAFzUNm1EM5/zVTR/Mjl9Lmd2lWummWaOb2ZXuWaaaeb4ZnaVO98zLd4HKMlxwMXAs6rq+gltqwOHAhtV1a5zObOrXDPNNHN8M7vKNdNMM8c3s6vceZ9ZA54SaCEvwLXAPadpvxdw7VzPXEiv1UwzzRzvXDPNNHN8MxfSax1lplNFDtZlwN2nad+y3WauZ3aVa6aZZo5vZle5Zppp5vhmdpU7rzO9YHWwPg18PsmBNOOeesc77QG8HvjQPMjsKtdMM80c38yucs0008zxzewqd35nDvrPIwt9af9xzgduBpa2y83tutfNl8yF9FrNNNPM8c4100wzxzdzIb3WUWV6weqQJNmcnmmCquqc+ZjZVa6ZZpo5vpld5Zppppnjm9lV7nzMtHiXJEmS5ggvWB2wJOsmeUySHZJkQtuaSd46HzK7yjXTTDPHN7OrXDPNNHN8M7vKndeZwxrftBAXYBvgAuAmmjFOi4G79LRvBCyd65kL6bWaaaaZ451rpplmjm/mQnqto8z0zPtgHQicBNwOuCNwNvDTJFvOs8yucs0008zxzewq10wzzRzfzK5y53fmoD9hLeSF5s5a20xY90HgPJq5P4fxSW/kmQvptZppppnjnWummWaOb+ZCeq2jzHSe98FaDbjVFcBV9ap23NPxwNPnSWZXuWaaaeb4ZnaVa6aZZo5vZle58zrT4n2wzgQWAaf3rqyqVyZZCfj2PMnsKtdMM80c38yucs0008zxzewqd15nOuZ9sL4F/MdkDVX1cuCLQCZrn2OZXeWaaaaZ45vZVa6ZZpo5vpld5c7rTOd5lyRJkuYIz7xLkiRJc4TFuyRJkjRHWLxLkiRJc4TFuyTpXyT5RpKlSXacon3Htv0bo+6bJC1kXrAqSfoXSTYETgOuAu5bVdf0tK0B/AZYm+amJJd000tJWng88y5J+hdVtQR4AXBX4L8mNL+3Xb/vqAr3JKskWX0UWZI0zizeJUmTqqrDgS8AL0qyG0CSXYAXA4dV1beTbJnkC0kuSHJjkj8neV+SNXv3lWSrJAcn+V2Sq5Jcm+SUJM+bmJvkgCSVZJskH0zyN+B6YLshv2RJGnveYVWSNJ2XAbsCn02yPfBZ4DzgZUm2BY4FLgc+1a6/b/uchyTZuar+0e5nF2An4HvAOcCawJOBTyfZsKoOnCT7f4HrgA/Q3Hb8gmG8QEmaSxzzLkmaVpKHAz8CLgHWBx5RVUcl+TWwGvDAqrqqZ/u9gG8C+1TVoe26NXvHzbfrVqIp/u8PbLCs0E9yAPA24MfA7lV103BfoSTNHQ6bkSRNq6qOBA4BNgA+3Rbu9wbuA3wJWC3JBssW4ETgGuDhPfvoveB19STrA+sBRwLrAFtNEv1hC3dJujWLd0nSTJw04evW7de3A0smLBfTDIvZaNmTk6yV5P1JzqUZCnNJu+27201uP0nmHwb5AiRpPnDMuyRpNtJ+/QDwwym2uazn+y8Be9KcwT8B+DuwFHg08EomP5l07UB6KknziMW7JGk2zmq/Lq2qo6fbMMm6NIX7F6rqhRPadh9O9yRpfnLYjCRpNk6luYnTC5NsMbExyW2SrNc+XLps9YRt7gD8y1SRkqSpeeZdktS3qqok/0kzW8xvknwW+B2wBnA34AnAG4FDq+qqJEcCz0xyHXAycBeam0CdQzODjSRpBizeJUmzUlW/SnJ/miL934EXAlcBfwYOBY7p2fyZwEHAY4Fn0Qy7eTPwD+BzI+u0JM1xzvMuSZIkzRGOeZckSZLmCIt3SZIkaY6weJckSZLmCIt3SZIkaY6weJckSZLmCIt3SZIkaY6weJckSZLmCIt3SZIkaY6weJckSZLmCIt3SZIkaY74/+CznLLNYXp0AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#prepare data from doi counts by years and visualise it\n",
    "published_papers = df.groupby('PubYear')[['DOI']].count().reset_index()\n",
    "published_papers['perc']=100*published_papers['DOI']/published_papers['DOI'].sum()\n",
    "\n",
    "ax = published_papers.plot(x='PubYear', y='perc', kind='bar', \n",
    "                                                figsize=(12, 6),\n",
    "                                                color = 'black', \n",
    "                                                fontsize = 14,\n",
    "                                                legend=False)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel(r'Papers published'\n",
    "           '\\n'\n",
    "           r'(percent from all in period 2000-2022)', fontsize=18)\n",
    "\n",
    "output_name = 'result_plots/published_papers.png'\n",
    "plt.savefig(output_name, dpi=300, bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "7a4c671a-e2cc-4103-a31e-ebb52eca8888",
   "metadata": {
    "id": "7a4c671a-e2cc-4103-a31e-ebb52eca8888"
   },
   "outputs": [],
   "source": [
    "#absolute values\n",
    "#df.groupby('PubYear')[['PubYear']].count().plot(kind='bar', \n",
    "#                                                figsize=(12, 6),\n",
    "#                                                color = 'black', \n",
    "#                                                fontsize = 14,\n",
    "#                                                legend=False)\n",
    "\n",
    "#plt.xlabel('Year', fontsize=18)\n",
    "#plt.ylabel('Papers published', fontsize=18)\n",
    "#plt.savefig('published_papers.png', dpi=300, bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1574ca1-39d2-4eae-a174-28237f4e91d2",
   "metadata": {
    "id": "a1574ca1-39d2-4eae-a174-28237f4e91d2"
   },
   "source": [
    "### Data cleaning \n",
    "In data cleaning step we \n",
    "* remove coding characters and punctiation symbols\n",
    "* set lowercases for the text of abstract"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "452f0e97-f28b-442e-9bf2-621226e68aff",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 166
    },
    "id": "452f0e97-f28b-442e-9bf2-621226e68aff",
    "outputId": "23f67494-6327-4216-d0e4-9991b083430a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Application of municipal biosolids in agriculture present a concern with potential uptake and bioaccumulation of pharmaceutical compounds from biosolids into agronomic plants. We evaluated the efficacy of biochar as a soil amendment to minimize uptake of antimicrobial agents (ciprofloxacin, triclocarban, and triclosan) in lettuce (Lactuca sativa) and carrot (Daucus carota) plants. Biochar reduced the concentration of ciprofloxacin and triclocarban in lettuce leaves and resulted in a 67% reduction of triclosan in carrot roots. There was no substantial difference in pharmaceutical concentrations in carrot and lettuce plant matter at low (2.0\\u2009g kg-1 soil) and high (20.4\\u2009g kg-1 soil) rates of applied biochar. The co-amendment of biochar and biosolids increased soil pH and nutrient content which were positively correlated with an increase in lettuce shoot biomass. Our results demonstrate the potential efficacy of using walnut shell biochar as a sorbent for pharmaceutical contaminants in soil without negatively affecting plant growth.'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#example of such characters \n",
    "df.Abstract[860]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e7c4fca4-e025-490b-af1a-daea4e143639",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 166
    },
    "id": "e7c4fca4-e025-490b-af1a-daea4e143639",
    "outputId": "05f7b4e7-839b-48d0-8587-2b659d1aea78"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'application of municipal biosolids in agriculture present a concern with potential uptake and bioaccumulation of pharmaceutical compounds from biosolids into agronomic plants we evaluated the efficacy of biochar as a soil amendment to minimize uptake of antimicrobial agents ciprofloxacin triclocarban and triclosan in lettuce lactuca sativa and carrot daucus carota plants biochar reduced the concentration of ciprofloxacin and triclocarban in lettuce leaves and resulted in a reduction of triclosan in carrot roots there was no substantial difference in pharmaceutical concentrations in carrot and lettuce plant matter at low g kg soil and high g kg soil rates of applied biochar the co amendment of biochar and biosolids increased soil ph and nutrient content which were positively correlated with an increase in lettuce shoot biomass our results demonstrate the potential efficacy of using walnut shell biochar as a sorbent for pharmaceutical contaminants in soil without negatively affecting plant growth '"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#apply encode-decode to the all items in column\n",
    "df.Abstract=df.Abstract.str.encode('utf-8').str.decode('ascii', 'ignore')\n",
    "\n",
    "#lowering letters, removing syntax and other non-letter characters \n",
    "df.Abstract = df.Abstract.str.replace('[^a-zA-Z]', ' ', regex=True)\n",
    "df.Abstract = df.Abstract.replace(r'\\s+', ' ', regex=True)\n",
    "\n",
    "\n",
    "#lower case\n",
    "df['Abstract'] = df['Abstract'].str.lower()\n",
    "\n",
    "#check\n",
    "df.Abstract[860]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccd86c44-17e5-4751-ac8f-8f618c019be3",
   "metadata": {
    "id": "ccd86c44-17e5-4751-ac8f-8f618c019be3"
   },
   "source": [
    "## Reading the data from database of Umwelt Bundesamt and comparing dimensions df data with it\n",
    "Database \"Pharmaceuticals in the environment\" (Excel file)\n",
    "<br> Sheet 1 -- concentrations found in environment with citation source from international and local litarature\n",
    "<br> Sheet 2 -- general information, including therapeutic groups descriptions\n",
    "https://www.umweltbundesamt.de/dokument/database-pharmaceuticals-in-the-environment-excel\n",
    "\n",
    "In this section we perform next steps:\n",
    "* we exctract name of analytes by which we perform search through the abstracts\n",
    "* we exctract corresponding therapeutic group and target group names and combine them with name of analytes into the dictionaries with keys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "8a7db87d-e7fd-43df-8025-8f8a49467e59",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 329
    },
    "id": "8a7db87d-e7fd-43df-8025-8f8a49467e59",
    "outputId": "bfa20670-a9d9-4941-9813-2898c72775e4"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name of Analyte</th>\n",
       "      <th>Analyte Name German</th>\n",
       "      <th>CAS number</th>\n",
       "      <th>Therapeutic Group English</th>\n",
       "      <th>Therapeutic Group German</th>\n",
       "      <th>Target Group</th>\n",
       "      <th>Type of Analyte</th>\n",
       "      <th>Pharma Parent Transformation</th>\n",
       "      <th>PharmaATC1</th>\n",
       "      <th>PharmaATC2</th>\n",
       "      <th>PharmaATC3</th>\n",
       "      <th>PharmaATC4</th>\n",
       "      <th>PharmaATC5</th>\n",
       "      <th>PharmaATC6</th>\n",
       "      <th>PharmaATC7</th>\n",
       "      <th>PharmaATC8</th>\n",
       "      <th>PharmaATC9</th>\n",
       "      <th>PharmaATC10</th>\n",
       "      <th>PharmaATC11</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1-(3-Chlorophenyl)-piperazine</td>\n",
       "      <td>1-(3-Chlorophenyl)-piperazin</td>\n",
       "      <td>6640-24-0</td>\n",
       "      <td>Antidepressant</td>\n",
       "      <td>Antidepressiva</td>\n",
       "      <td>Human</td>\n",
       "      <td>Transformation product</td>\n",
       "      <td>Trazodone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1,2-Dihydro budesonide</td>\n",
       "      <td>1,2-Dihydro budesonide</td>\n",
       "      <td>137174-25-5</td>\n",
       "      <td>Glucocorticoids</td>\n",
       "      <td>Glucocorticode</td>\n",
       "      <td>Human</td>\n",
       "      <td>Transformation product</td>\n",
       "      <td>Budesonide</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1,2-Dihydro clobetasol propionate</td>\n",
       "      <td>1,2-Dihydro clobetasol propionate</td>\n",
       "      <td>25120-99-4</td>\n",
       "      <td>Glucocorticoids</td>\n",
       "      <td>Glucocorticode</td>\n",
       "      <td>Human</td>\n",
       "      <td>Transformation product</td>\n",
       "      <td>Clobetasol</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Name of Analyte                Analyte Name German  \\\n",
       "0      1-(3-Chlorophenyl)-piperazine       1-(3-Chlorophenyl)-piperazin   \n",
       "1             1,2-Dihydro budesonide             1,2-Dihydro budesonide   \n",
       "2  1,2-Dihydro clobetasol propionate  1,2-Dihydro clobetasol propionate   \n",
       "\n",
       "    CAS number Therapeutic Group English Therapeutic Group German  \\\n",
       "0    6640-24-0            Antidepressant           Antidepressiva   \n",
       "1  137174-25-5           Glucocorticoids           Glucocorticode   \n",
       "2   25120-99-4           Glucocorticoids           Glucocorticode   \n",
       "\n",
       "  Target Group         Type of Analyte Pharma Parent Transformation  \\\n",
       "0        Human  Transformation product                    Trazodone   \n",
       "1        Human  Transformation product                   Budesonide   \n",
       "2        Human  Transformation product                   Clobetasol   \n",
       "\n",
       "  PharmaATC1 PharmaATC2 PharmaATC3 PharmaATC4 PharmaATC5 PharmaATC6  \\\n",
       "0        NaN        NaN        NaN        NaN        NaN        NaN   \n",
       "1        NaN        NaN        NaN        NaN        NaN        NaN   \n",
       "2        NaN        NaN        NaN        NaN        NaN        NaN   \n",
       "\n",
       "  PharmaATC7 PharmaATC8 PharmaATC9 PharmaATC10 PharmaATC11  \n",
       "0        NaN        NaN        NaN         NaN         NaN  \n",
       "1        NaN        NaN        NaN         NaN         NaN  \n",
       "2        NaN        NaN        NaN         NaN         NaN  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FOLDER = './data/'\n",
    "path2 = os.path.join(FOLDER, 'pharms-uba_v3_2021_0.xlsx')\n",
    "source_base = pd.read_excel(path2, sheet_name = 'Pharma_Agents_v3')\n",
    "source_base.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "442c13f0-10a0-4bbb-b342-70a4ef27d93a",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "442c13f0-10a0-4bbb-b342-70a4ef27d93a",
    "outputId": "457c1cda-388d-4577-e798-433004247286"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1-(3-chlorophenyl)-piperazine',\n",
       " '1,2-dihydro budesonide',\n",
       " '1,2-dihydro clobetasol propionate',\n",
       " '1,2-dihydro fluocinolone acetonide',\n",
       " '1,5-dimethyl-1,2-dehydro-3-pyrazolone',\n",
       " '10,11-dihydro-10,11-dihydroxy carbamazepine',\n",
       " '10,11-dihydro-10,11-epoxycarbamazepine',\n",
       " '10,11-dihydro-10-hydroxycarbamazepine',\n",
       " '10,11-dihydrocarbamazepine',\n",
       " '10-hydroxy-amitriptyline']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#extracting of unique names of pharm analytes from UBA database\n",
    "source_list = source_base['Name of Analyte'].str.lower().unique().tolist()\n",
    "#looking at the list\n",
    "source_list[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d0b4548d-f119-4016-b45a-ac6afeaba538",
   "metadata": {
    "id": "d0b4548d-f119-4016-b45a-ac6afeaba538",
    "tags": []
   },
   "outputs": [],
   "source": [
    "#creating keys dictionaries\n",
    "source_base = source_base.apply(lambda x: x.astype(str).str.lower())\n",
    "\n",
    "#rename few columns for convenience\n",
    "source_base.rename(columns = {'Name of Analyte':'name_of_analyte', \n",
    "                              'Therapeutic Group English':'therapeutic_group_english',\n",
    "                              'Target Group': 'target_group'}, inplace = True)\n",
    "\n",
    "#setting a dictionary with pair key:value == name of analyte:therapeutic group\n",
    "therapeutic_group_keys = dict (zip(source_base['name_of_analyte'], source_base['therapeutic_group_english']))\n",
    "therapeutic_group_keys.update({'acetaminophen':'analgesics',\n",
    "                              'paracetamol':'analgesics',\n",
    "                              'aspirin':'nonsteroidal anti-inflammatory drugs',\n",
    "                              'acetylsalicylic acid':'nonsteroidal anti-inflammatory drugs', \n",
    "                              'clofibric acid': 'lipid-lowering drugs',\n",
    "                              'salicylic acid': 'skin care',\n",
    "                              'gemfibrocil':'lipid-lowering drugs', \n",
    "                              'flufenamic acid': 'nonsteroidal anti-inflammatory drugs',\n",
    "                              'caffeine':'other'\n",
    "                              })\n",
    "\n",
    "#setting a dictionary with pair key:value == name of analyte:target group\n",
    "target_group_keys = dict (zip(source_base['name_of_analyte'], source_base['target_group']))\n",
    "target_group_keys.update({'acetaminophen':'human',\n",
    "                          'paracetamol':'human',\n",
    "                          'aspirin':'human and veterinary',\n",
    "                          'acetylsalicylic acid':'human and veterinary',\n",
    "                          'clofibric acid': 'human',\n",
    "                          'salicylic acid': 'human',\n",
    "                          'gemfibrocil':'human',\n",
    "                          'flufenamic acid': 'human',\n",
    "                          'caffeine':'human'})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d15997dc-5b00-4206-8ec9-1fd613d736f7",
   "metadata": {
    "id": "d15997dc-5b00-4206-8ec9-1fd613d736f7"
   },
   "source": [
    "## List of mentioned pharmaceuticals in the abstract (unique)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0f0e694-79f9-481c-8546-8536a8de6abb",
   "metadata": {
    "id": "e0f0e694-79f9-481c-8546-8536a8de6abb"
   },
   "source": [
    "We then use the ``source list`` to search unique mentions of analytes in the abstract texts to extract occurring words and estimate their occurrence level. The search is based on the unique mention of analyte in the abstracts, new column ``pharmaceuticals`` in ``df`` contains lists of unique words, even if they can be mentioned multiple times in the corresponding abstract."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "2fed365f-67c7-4ce7-bf8c-242d408b10e2",
   "metadata": {
    "id": "2fed365f-67c7-4ce7-bf8c-242d408b10e2"
   },
   "outputs": [],
   "source": [
    "#additional words and phrases for search\n",
    "add_words_list = ['acetaminophen',\n",
    "                 'paracetamol',\n",
    "                 'aspirin',\n",
    "                 'acetylsalicylic acid',\n",
    "                 'clofibric acid',\n",
    "                 'salicylic acid',\n",
    "                 'gemfibrocil',\n",
    "                 'flufenamic acid',\n",
    "                 'caffeine']\n",
    "\n",
    "source_list = source_list+add_words_list\n",
    "\n",
    "\n",
    "df['pharmaceuticals'] = 'NaN' #new column\n",
    "\n",
    "#filling new column with unique words mathching with source list from UBA\n",
    "for i in range(len(df)):\n",
    "    # select i row in df\n",
    "    select = df.iloc[i,:]\n",
    "    # select only Abstract from row\n",
    "    abstract = select['Abstract']\n",
    "    # sometimes abstract it's NaN, in such case return NaN \n",
    "    if type(abstract) != str:\n",
    "        df.at[i, \"pharmaceuticals\"] = 'NaN'\n",
    "    else:\n",
    "        # empty set to collect unique names of pharmaceuticals\n",
    "        list_of_target_words = set() #for unique values\n",
    "        for item in source_list:\n",
    "            # if pharmaceutical not find in source_list - > continue to next \n",
    "            if abstract.find(item)==-1:\n",
    "                continue\n",
    "            else:\n",
    "                list_of_target_words.add(item)\n",
    "        #for unique sorted names\n",
    "        df.at[i, \"pharmaceuticals\"] = list(sorted(list_of_target_words)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "51e21a27-94ed-4441-acc3-9f287b29d8d9",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "51e21a27-94ed-4441-acc3-9f287b29d8d9",
    "outputId": "3ebcd1ac-07ad-41b4-9dcc-fcb55aeb94ae"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    [acetaminophen, aspirin, carbamazepine, chlorp...\n",
       "1             [amprolium, carbadox, monensin, tylosin]\n",
       "2    [carbamazepine, clindamycin, diltiazem, diphen...\n",
       "3    [carbamazepine, fluoxetine, naproxen, sulfamet...\n",
       "4                                                   []\n",
       "5                            [caffeine, carbamazepine]\n",
       "6    [ciprofloxacin, clofibric acid, diclofenac, ib...\n",
       "7    [acetaminophen, carbamazepine, codeine, ibupro...\n",
       "8    [bezafibrate, clarithromycin, clindamycin, dic...\n",
       "9                                                   []\n",
       "Name: pharmaceuticals, dtype: object"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#look at the result -- new column in the df dataset \n",
    "df.pharmaceuticals.head(10)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "7aa451ef-ef26-409b-9c08-99b94ae2e611",
   "metadata": {
    "id": "7aa451ef-ef26-409b-9c08-99b94ae2e611"
   },
   "outputs": [],
   "source": [
    "#skipped step: manual catching of the missed PPCP names on a random sample of length 20\n",
    "#random_sample = df_analysis.iloc[:,[6,32]].sample(20).reset_index(drop=True)\n",
    "\n",
    "#for i, row in random_sample.iterrows():\n",
    "#    print(row['pharmaceuticals'])\n",
    "#    print(row['Abstract'])\n",
    "#    print() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "9c453442-d229-4078-9f1c-008c8b86dfb0",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9c453442-d229-4078-9f1c-008c8b86dfb0",
    "outputId": "30083e9e-9a20-43f1-a75d-d0a71364bb3f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30.8 % of initial entries of new column are not 0 and non-NaN\n"
     ]
    }
   ],
   "source": [
    "#check how many entries of new column \"pharmaceuticals\" are not 0 and not NaNs\n",
    "print(round(100*(len(df[df[\"pharmaceuticals\"].str.len(\n",
    ") != 0])+len(df.query(\"pharmaceuticals == 'NaN'\")))/len(df),1), '% of initial entries of new column are not 0 and non-NaN')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4958125b-3e5e-40f6-acf7-034e10944dcb",
   "metadata": {
    "id": "4958125b-3e5e-40f6-acf7-034e10944dcb"
   },
   "source": [
    "## Final dataframe for the analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5ad2706-3a31-4b06-972d-79f21ae35762",
   "metadata": {
    "id": "a5ad2706-3a31-4b06-972d-79f21ae35762"
   },
   "source": [
    "For the further work we will use new dataframe ``df_analysis``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "2f5d7192-447f-4078-a895-a3475dd510c2",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2f5d7192-447f-4078-a895-a3475dd510c2",
    "outputId": "5cb88771-12d1-453e-af8f-6bc2fa0cdd19"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "843"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#removing NaN and zero data \n",
    "df = df.query(\"pharmaceuticals != 'NaN'\")\n",
    "\n",
    "df_analysis = df[df[\"pharmaceuticals\"].str.len() != 0]\n",
    "len(df_analysis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "2e2e9c50-9a12-452f-a128-c0ffa097e95e",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 445
    },
    "id": "2e2e9c50-9a12-452f-a128-c0ffa097e95e",
    "outputId": "ec6d89d5-affd-4d5c-d9ac-92901b39abfe"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Rank</th>\n",
       "      <th>Publication ID</th>\n",
       "      <th>DOI</th>\n",
       "      <th>PMID</th>\n",
       "      <th>PMCID</th>\n",
       "      <th>Title</th>\n",
       "      <th>Abstract</th>\n",
       "      <th>Acknowledgements</th>\n",
       "      <th>Source title</th>\n",
       "      <th>Anthology title</th>\n",
       "      <th>...</th>\n",
       "      <th>Authors Affiliations</th>\n",
       "      <th>Times cited</th>\n",
       "      <th>Recent citations</th>\n",
       "      <th>RCR</th>\n",
       "      <th>FCR</th>\n",
       "      <th>Source Linkout</th>\n",
       "      <th>Dimensions URL</th>\n",
       "      <th>FOR (ANZSRC) Categories</th>\n",
       "      <th>Sustainable Development Goals</th>\n",
       "      <th>pharmaceuticals</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3405</td>\n",
       "      <td>pub.1034753825</td>\n",
       "      <td>10.1002/etc.487</td>\n",
       "      <td>21312249.0</td>\n",
       "      <td>nan</td>\n",
       "      <td>influence of sewage and pharmaceuticals on soi...</td>\n",
       "      <td>although sewage effluent application to land i...</td>\n",
       "      <td>we thank catriona macdonald and esr for advice...</td>\n",
       "      <td>environmental toxicology and chemistry</td>\n",
       "      <td>nan</td>\n",
       "      <td>...</td>\n",
       "      <td>gielen, gerty j. h. p. (scion); clinton, peter...</td>\n",
       "      <td>14</td>\n",
       "      <td>3</td>\n",
       "      <td>0.43</td>\n",
       "      <td>1.47</td>\n",
       "      <td>nan</td>\n",
       "      <td>https://app.dimensions.ai/details/publication/...</td>\n",
       "      <td>0605 microbiology; 0503 soil sciences; 05 envi...</td>\n",
       "      <td>nan</td>\n",
       "      <td>[acetaminophen, aspirin, carbamazepine, chlorp...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3376</td>\n",
       "      <td>pub.1069010127</td>\n",
       "      <td>10.2134/jeq2009.0090</td>\n",
       "      <td>20830908.0</td>\n",
       "      <td>nan</td>\n",
       "      <td>selected veterinary pharmaceuticals in agricul...</td>\n",
       "      <td>veterinary pharmaceuticals are commonly admini...</td>\n",
       "      <td>this study was supported in part by agricultur...</td>\n",
       "      <td>journal of environmental quality</td>\n",
       "      <td>nan</td>\n",
       "      <td>...</td>\n",
       "      <td>song, wenlu (michigan state university); ding,...</td>\n",
       "      <td>48</td>\n",
       "      <td>3</td>\n",
       "      <td>1.74</td>\n",
       "      <td>4.91</td>\n",
       "      <td>nan</td>\n",
       "      <td>https://app.dimensions.ai/details/publication/...</td>\n",
       "      <td>0503 soil sciences; 05 environmental sciences</td>\n",
       "      <td>nan</td>\n",
       "      <td>[amprolium, carbadox, monensin, tylosin]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3320</td>\n",
       "      <td>pub.1026757075</td>\n",
       "      <td>10.1007/s00244-010-9500-y</td>\n",
       "      <td>20340015.0</td>\n",
       "      <td>nan</td>\n",
       "      <td>dissipation and leaching potential of selected...</td>\n",
       "      <td>biosolids land application is an important pat...</td>\n",
       "      <td>this research was funded by the usda-csrees pr...</td>\n",
       "      <td>archives of environmental contamination and to...</td>\n",
       "      <td>nan</td>\n",
       "      <td>...</td>\n",
       "      <td>wu, chenxi (university of toledo); spongberg, ...</td>\n",
       "      <td>38</td>\n",
       "      <td>4</td>\n",
       "      <td>1.2</td>\n",
       "      <td>3.97</td>\n",
       "      <td>nan</td>\n",
       "      <td>https://app.dimensions.ai/details/publication/...</td>\n",
       "      <td>0503 soil sciences; 05 environmental sciences</td>\n",
       "      <td>nan</td>\n",
       "      <td>[carbamazepine, clindamycin, diltiazem, diphen...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 33 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Rank  Publication ID                        DOI        PMID PMCID  \\\n",
       "0  3405  pub.1034753825            10.1002/etc.487  21312249.0   nan   \n",
       "1  3376  pub.1069010127       10.2134/jeq2009.0090  20830908.0   nan   \n",
       "2  3320  pub.1026757075  10.1007/s00244-010-9500-y  20340015.0   nan   \n",
       "\n",
       "                                               Title  \\\n",
       "0  influence of sewage and pharmaceuticals on soi...   \n",
       "1  selected veterinary pharmaceuticals in agricul...   \n",
       "2  dissipation and leaching potential of selected...   \n",
       "\n",
       "                                            Abstract  \\\n",
       "0  although sewage effluent application to land i...   \n",
       "1  veterinary pharmaceuticals are commonly admini...   \n",
       "2  biosolids land application is an important pat...   \n",
       "\n",
       "                                    Acknowledgements  \\\n",
       "0  we thank catriona macdonald and esr for advice...   \n",
       "1  this study was supported in part by agricultur...   \n",
       "2  this research was funded by the usda-csrees pr...   \n",
       "\n",
       "                                        Source title Anthology title  ...  \\\n",
       "0             environmental toxicology and chemistry             nan  ...   \n",
       "1                   journal of environmental quality             nan  ...   \n",
       "2  archives of environmental contamination and to...             nan  ...   \n",
       "\n",
       "                                Authors Affiliations Times cited  \\\n",
       "0  gielen, gerty j. h. p. (scion); clinton, peter...          14   \n",
       "1  song, wenlu (michigan state university); ding,...          48   \n",
       "2  wu, chenxi (university of toledo); spongberg, ...          38   \n",
       "\n",
       "   Recent citations   RCR   FCR Source Linkout  \\\n",
       "0                 3  0.43  1.47            nan   \n",
       "1                 3  1.74  4.91            nan   \n",
       "2                 4   1.2  3.97            nan   \n",
       "\n",
       "                                      Dimensions URL  \\\n",
       "0  https://app.dimensions.ai/details/publication/...   \n",
       "1  https://app.dimensions.ai/details/publication/...   \n",
       "2  https://app.dimensions.ai/details/publication/...   \n",
       "\n",
       "                             FOR (ANZSRC) Categories  \\\n",
       "0  0605 microbiology; 0503 soil sciences; 05 envi...   \n",
       "1      0503 soil sciences; 05 environmental sciences   \n",
       "2      0503 soil sciences; 05 environmental sciences   \n",
       "\n",
       "  Sustainable Development Goals  \\\n",
       "0                           nan   \n",
       "1                           nan   \n",
       "2                           nan   \n",
       "\n",
       "                                     pharmaceuticals  \n",
       "0  [acetaminophen, aspirin, carbamazepine, chlorp...  \n",
       "1           [amprolium, carbadox, monensin, tylosin]  \n",
       "2  [carbamazepine, clindamycin, diltiazem, diphen...  \n",
       "\n",
       "[3 rows x 33 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#lowercase for main columns \n",
    "df_analysis = df_analysis.apply(lambda x: x.astype(str).str.lower(\n",
    ") if x.name != 'pharmaceuticals' and x.name != 'Times cited' and x.name != 'PubYear' else x )\n",
    "\n",
    "display(df_analysis.head(3))\n",
    "#print(df_analysis.info())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01ad2428-3bd8-4556-b8b4-d7d160da6091",
   "metadata": {
    "id": "01ad2428-3bd8-4556-b8b4-d7d160da6091"
   },
   "source": [
    "### Manual excluding of the irrelevant environmental matrix (not soils)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "d6f1d3d6-b163-4077-ae4e-ca6ef64c3fa5",
   "metadata": {
    "id": "d6f1d3d6-b163-4077-ae4e-ca6ef64c3fa5"
   },
   "outputs": [],
   "source": [
    "#check the list of publications on the scope: is the paper really not about the groundwater,\n",
    "#and about soils themselves\n",
    "#df_analysis[df_analysis.Title.str.contains(\"groundwater\")].to_csv('groundwater_papers.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "791ce9be-eb24-48a2-a2a7-727fead98f47",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 143
    },
    "id": "791ce9be-eb24-48a2-a2a7-727fead98f47",
    "outputId": "9bd4e966-6970-4bb8-ce50-595a477ca76d"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DOI</th>\n",
       "      <th>Check</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10.1016/j.chemosphere.2011.12.018</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.2166/wst.2012.290</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10.1016/j.scitotenv.2012.06.031</td>\n",
       "      <td>exclude</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 DOI    Check\n",
       "0  10.1016/j.chemosphere.2011.12.018       ok\n",
       "1               10.2166/wst.2012.290       ok\n",
       "2    10.1016/j.scitotenv.2012.06.031  exclude"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FOLDER = './data/'\n",
    "path_3 = os.path.join(FOLDER, 'groundwater_papers_manually_checked.csv')\n",
    "doi_to_remove = pd.read_csv(path_3)\n",
    "doi_to_remove.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "c4a87bd6-f148-4409-a934-391048504204",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "c4a87bd6-f148-4409-a934-391048504204",
    "outputId": "47972695-41b6-4fc4-e6ad-e3adf9146fe0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "828"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#excluding of irrelevant publications\n",
    "doi_exclude = doi_to_remove.query(\"Check == 'exclude'\")['DOI'].to_list()\n",
    "df_analysis = df_analysis.query(\"DOI != @doi_exclude\")\n",
    "#number of final entries\n",
    "len(df_analysis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "2a2e796e-bf22-42f1-91b7-ab28fb8b08dd",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "2a2e796e-bf22-42f1-91b7-ab28fb8b08dd",
    "outputId": "3e4416c6-bc31-4380-89d5-8c41a0f9e98b"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Rank</th>\n",
       "      <th>Publication ID</th>\n",
       "      <th>DOI</th>\n",
       "      <th>PMID</th>\n",
       "      <th>PMCID</th>\n",
       "      <th>Title</th>\n",
       "      <th>Abstract</th>\n",
       "      <th>Acknowledgements</th>\n",
       "      <th>Source title</th>\n",
       "      <th>Anthology title</th>\n",
       "      <th>...</th>\n",
       "      <th>Authors Affiliations</th>\n",
       "      <th>Times cited</th>\n",
       "      <th>Recent citations</th>\n",
       "      <th>RCR</th>\n",
       "      <th>FCR</th>\n",
       "      <th>Source Linkout</th>\n",
       "      <th>Dimensions URL</th>\n",
       "      <th>FOR (ANZSRC) Categories</th>\n",
       "      <th>Sustainable Development Goals</th>\n",
       "      <th>pharmaceuticals</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3405</td>\n",
       "      <td>pub.1034753825</td>\n",
       "      <td>10.1002/etc.487</td>\n",
       "      <td>21312249.0</td>\n",
       "      <td>nan</td>\n",
       "      <td>influence of sewage and pharmaceuticals on soi...</td>\n",
       "      <td>although sewage effluent application to land i...</td>\n",
       "      <td>we thank catriona macdonald and esr for advice...</td>\n",
       "      <td>environmental toxicology and chemistry</td>\n",
       "      <td>nan</td>\n",
       "      <td>...</td>\n",
       "      <td>gielen, gerty j. h. p. (scion); clinton, peter...</td>\n",
       "      <td>14</td>\n",
       "      <td>3</td>\n",
       "      <td>0.43</td>\n",
       "      <td>1.47</td>\n",
       "      <td>nan</td>\n",
       "      <td>https://app.dimensions.ai/details/publication/...</td>\n",
       "      <td>0605 microbiology; 0503 soil sciences; 05 envi...</td>\n",
       "      <td>nan</td>\n",
       "      <td>[acetaminophen, aspirin, carbamazepine, chlorp...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3376</td>\n",
       "      <td>pub.1069010127</td>\n",
       "      <td>10.2134/jeq2009.0090</td>\n",
       "      <td>20830908.0</td>\n",
       "      <td>nan</td>\n",
       "      <td>selected veterinary pharmaceuticals in agricul...</td>\n",
       "      <td>veterinary pharmaceuticals are commonly admini...</td>\n",
       "      <td>this study was supported in part by agricultur...</td>\n",
       "      <td>journal of environmental quality</td>\n",
       "      <td>nan</td>\n",
       "      <td>...</td>\n",
       "      <td>song, wenlu (michigan state university); ding,...</td>\n",
       "      <td>48</td>\n",
       "      <td>3</td>\n",
       "      <td>1.74</td>\n",
       "      <td>4.91</td>\n",
       "      <td>nan</td>\n",
       "      <td>https://app.dimensions.ai/details/publication/...</td>\n",
       "      <td>0503 soil sciences; 05 environmental sciences</td>\n",
       "      <td>nan</td>\n",
       "      <td>[amprolium, carbadox, monensin, tylosin]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3320</td>\n",
       "      <td>pub.1026757075</td>\n",
       "      <td>10.1007/s00244-010-9500-y</td>\n",
       "      <td>20340015.0</td>\n",
       "      <td>nan</td>\n",
       "      <td>dissipation and leaching potential of selected...</td>\n",
       "      <td>biosolids land application is an important pat...</td>\n",
       "      <td>this research was funded by the usda-csrees pr...</td>\n",
       "      <td>archives of environmental contamination and to...</td>\n",
       "      <td>nan</td>\n",
       "      <td>...</td>\n",
       "      <td>wu, chenxi (university of toledo); spongberg, ...</td>\n",
       "      <td>38</td>\n",
       "      <td>4</td>\n",
       "      <td>1.2</td>\n",
       "      <td>3.97</td>\n",
       "      <td>nan</td>\n",
       "      <td>https://app.dimensions.ai/details/publication/...</td>\n",
       "      <td>0503 soil sciences; 05 environmental sciences</td>\n",
       "      <td>nan</td>\n",
       "      <td>[carbamazepine, clindamycin, diltiazem, diphen...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3304</td>\n",
       "      <td>pub.1015133810</td>\n",
       "      <td>10.1897/08-657.1</td>\n",
       "      <td>19580336.0</td>\n",
       "      <td>nan</td>\n",
       "      <td>factors affecting the degradation of pharmaceu...</td>\n",
       "      <td>pharmaceuticals may be released to the soil en...</td>\n",
       "      <td>the development of this paper was funded by th...</td>\n",
       "      <td>environmental toxicology and chemistry</td>\n",
       "      <td>nan</td>\n",
       "      <td>...</td>\n",
       "      <td>monteiro, sara c. (university of york; fera sc...</td>\n",
       "      <td>133</td>\n",
       "      <td>14</td>\n",
       "      <td>4.51</td>\n",
       "      <td>13.63</td>\n",
       "      <td>https://setac.onlinelibrary.wiley.com/doi/pdfd...</td>\n",
       "      <td>https://app.dimensions.ai/details/publication/...</td>\n",
       "      <td>0503 soil sciences; 05 environmental sciences;...</td>\n",
       "      <td>nan</td>\n",
       "      <td>[carbamazepine, fluoxetine, naproxen, sulfamet...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3174</td>\n",
       "      <td>pub.1036761509</td>\n",
       "      <td>10.1016/j.chemosphere.2008.09.048</td>\n",
       "      <td>18996568.0</td>\n",
       "      <td>nan</td>\n",
       "      <td>factors impacting on pharmaceutical leaching f...</td>\n",
       "      <td>sewage effluent application to land is a treat...</td>\n",
       "      <td>we thank the new zealand foundation for resear...</td>\n",
       "      <td>chemosphere</td>\n",
       "      <td>nan</td>\n",
       "      <td>...</td>\n",
       "      <td>gielen, gerty j.h.p. (scion); van den heuvel, ...</td>\n",
       "      <td>44</td>\n",
       "      <td>5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.97</td>\n",
       "      <td>nan</td>\n",
       "      <td>https://app.dimensions.ai/details/publication/...</td>\n",
       "      <td>0503 soil sciences; 05 environmental sciences</td>\n",
       "      <td>nan</td>\n",
       "      <td>[caffeine, carbamazepine]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>3129</td>\n",
       "      <td>pub.1041971931</td>\n",
       "      <td>10.1007/s00216-011-4826-5</td>\n",
       "      <td>21416164.0</td>\n",
       "      <td>nan</td>\n",
       "      <td>assessment of the occurrence and distribution ...</td>\n",
       "      <td>the distribution of pharmaceuticals between wa...</td>\n",
       "      <td>this work has been supported by the spanish mi...</td>\n",
       "      <td>analytical and bioanalytical chemistry</td>\n",
       "      <td>nan</td>\n",
       "      <td>...</td>\n",
       "      <td>vazquez-roig, pablo (university of valencia); ...</td>\n",
       "      <td>74</td>\n",
       "      <td>9</td>\n",
       "      <td>3.08</td>\n",
       "      <td>8.9</td>\n",
       "      <td>nan</td>\n",
       "      <td>https://app.dimensions.ai/details/publication/...</td>\n",
       "      <td>03 chemical sciences; 0301 analytical chemistry</td>\n",
       "      <td>nan</td>\n",
       "      <td>[ciprofloxacin, clofibric acid, diclofenac, ib...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3107</td>\n",
       "      <td>pub.1005223864</td>\n",
       "      <td>10.1016/j.scitotenv.2012.08.036</td>\n",
       "      <td>23021792.0</td>\n",
       "      <td>nan</td>\n",
       "      <td>risk assessment on the presence of pharmaceuti...</td>\n",
       "      <td>this study is focused on the occurrence of pha...</td>\n",
       "      <td>this work has been supported by the spanish mi...</td>\n",
       "      <td>the science of the total environment</td>\n",
       "      <td>nan</td>\n",
       "      <td>...</td>\n",
       "      <td>vazquez-roig, pablo (university of valencia); ...</td>\n",
       "      <td>142</td>\n",
       "      <td>30</td>\n",
       "      <td>5.78</td>\n",
       "      <td>16.17</td>\n",
       "      <td>nan</td>\n",
       "      <td>https://app.dimensions.ai/details/publication/...</td>\n",
       "      <td>09 engineering; 0503 soil sciences; 03 chemica...</td>\n",
       "      <td>nan</td>\n",
       "      <td>[acetaminophen, carbamazepine, codeine, ibupro...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3097</td>\n",
       "      <td>pub.1069010134</td>\n",
       "      <td>10.2134/jeq2009.0105</td>\n",
       "      <td>20830907.0</td>\n",
       "      <td>nan</td>\n",
       "      <td>transport of pharmaceuticals in columns of a w...</td>\n",
       "      <td>the irrigation or agricultural land with waste...</td>\n",
       "      <td>j. siemens, g. huschek, and g. walshe conducte...</td>\n",
       "      <td>journal of environmental quality</td>\n",
       "      <td>nan</td>\n",
       "      <td>...</td>\n",
       "      <td>siemens, jan (university of bonn); huschek, ge...</td>\n",
       "      <td>28</td>\n",
       "      <td>4</td>\n",
       "      <td>0.92</td>\n",
       "      <td>2.92</td>\n",
       "      <td>nan</td>\n",
       "      <td>https://app.dimensions.ai/details/publication/...</td>\n",
       "      <td>0503 soil sciences; 05 environmental sciences</td>\n",
       "      <td>nan</td>\n",
       "      <td>[bezafibrate, clarithromycin, clindamycin, dic...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>3072</td>\n",
       "      <td>pub.1038033252</td>\n",
       "      <td>10.1016/j.talanta.2011.11.054</td>\n",
       "      <td>22265554.0</td>\n",
       "      <td>nan</td>\n",
       "      <td>ultrasonic or accelerated solvent extraction f...</td>\n",
       "      <td>different veterinary pharmaceuticals are used ...</td>\n",
       "      <td>this project was financially supported by the ...</td>\n",
       "      <td>talanta</td>\n",
       "      <td>nan</td>\n",
       "      <td>...</td>\n",
       "      <td>chitescu, carmen lidia (\"dunarea de jos\" unive...</td>\n",
       "      <td>49</td>\n",
       "      <td>7</td>\n",
       "      <td>1.83</td>\n",
       "      <td>5.77</td>\n",
       "      <td>nan</td>\n",
       "      <td>https://app.dimensions.ai/details/publication/...</td>\n",
       "      <td>03 chemical sciences; 0301 analytical chemistry</td>\n",
       "      <td>nan</td>\n",
       "      <td>[oxytetracycline, tetracycline]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>3072</td>\n",
       "      <td>pub.1034164273</td>\n",
       "      <td>10.1016/j.talanta.2009.01.035</td>\n",
       "      <td>19269487.0</td>\n",
       "      <td>nan</td>\n",
       "      <td>the analysis of a group of acidic pharmaceutic...</td>\n",
       "      <td>the analysis of pharmaceuticals and potential ...</td>\n",
       "      <td>we would like to acknowledge the secretaría de...</td>\n",
       "      <td>talanta</td>\n",
       "      <td>nan</td>\n",
       "      <td>...</td>\n",
       "      <td>durán-alvarez, juan c. (national autonomous un...</td>\n",
       "      <td>118</td>\n",
       "      <td>14</td>\n",
       "      <td>3.97</td>\n",
       "      <td>11.02</td>\n",
       "      <td>nan</td>\n",
       "      <td>https://app.dimensions.ai/details/publication/...</td>\n",
       "      <td>03 chemical sciences; 0301 analytical chemistry</td>\n",
       "      <td>nan</td>\n",
       "      <td>[carbamazepine, estradiol, estrone]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 33 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Rank  Publication ID                                DOI        PMID PMCID  \\\n",
       "0  3405  pub.1034753825                    10.1002/etc.487  21312249.0   nan   \n",
       "1  3376  pub.1069010127               10.2134/jeq2009.0090  20830908.0   nan   \n",
       "2  3320  pub.1026757075          10.1007/s00244-010-9500-y  20340015.0   nan   \n",
       "3  3304  pub.1015133810                   10.1897/08-657.1  19580336.0   nan   \n",
       "4  3174  pub.1036761509  10.1016/j.chemosphere.2008.09.048  18996568.0   nan   \n",
       "5  3129  pub.1041971931          10.1007/s00216-011-4826-5  21416164.0   nan   \n",
       "6  3107  pub.1005223864    10.1016/j.scitotenv.2012.08.036  23021792.0   nan   \n",
       "7  3097  pub.1069010134               10.2134/jeq2009.0105  20830907.0   nan   \n",
       "8  3072  pub.1038033252      10.1016/j.talanta.2011.11.054  22265554.0   nan   \n",
       "9  3072  pub.1034164273      10.1016/j.talanta.2009.01.035  19269487.0   nan   \n",
       "\n",
       "                                               Title  \\\n",
       "0  influence of sewage and pharmaceuticals on soi...   \n",
       "1  selected veterinary pharmaceuticals in agricul...   \n",
       "2  dissipation and leaching potential of selected...   \n",
       "3  factors affecting the degradation of pharmaceu...   \n",
       "4  factors impacting on pharmaceutical leaching f...   \n",
       "5  assessment of the occurrence and distribution ...   \n",
       "6  risk assessment on the presence of pharmaceuti...   \n",
       "7  transport of pharmaceuticals in columns of a w...   \n",
       "8  ultrasonic or accelerated solvent extraction f...   \n",
       "9  the analysis of a group of acidic pharmaceutic...   \n",
       "\n",
       "                                            Abstract  \\\n",
       "0  although sewage effluent application to land i...   \n",
       "1  veterinary pharmaceuticals are commonly admini...   \n",
       "2  biosolids land application is an important pat...   \n",
       "3  pharmaceuticals may be released to the soil en...   \n",
       "4  sewage effluent application to land is a treat...   \n",
       "5  the distribution of pharmaceuticals between wa...   \n",
       "6  this study is focused on the occurrence of pha...   \n",
       "7  the irrigation or agricultural land with waste...   \n",
       "8  different veterinary pharmaceuticals are used ...   \n",
       "9  the analysis of pharmaceuticals and potential ...   \n",
       "\n",
       "                                    Acknowledgements  \\\n",
       "0  we thank catriona macdonald and esr for advice...   \n",
       "1  this study was supported in part by agricultur...   \n",
       "2  this research was funded by the usda-csrees pr...   \n",
       "3  the development of this paper was funded by th...   \n",
       "4  we thank the new zealand foundation for resear...   \n",
       "5  this work has been supported by the spanish mi...   \n",
       "6  this work has been supported by the spanish mi...   \n",
       "7  j. siemens, g. huschek, and g. walshe conducte...   \n",
       "8  this project was financially supported by the ...   \n",
       "9  we would like to acknowledge the secretaría de...   \n",
       "\n",
       "                                        Source title Anthology title  ...  \\\n",
       "0             environmental toxicology and chemistry             nan  ...   \n",
       "1                   journal of environmental quality             nan  ...   \n",
       "2  archives of environmental contamination and to...             nan  ...   \n",
       "3             environmental toxicology and chemistry             nan  ...   \n",
       "4                                        chemosphere             nan  ...   \n",
       "5             analytical and bioanalytical chemistry             nan  ...   \n",
       "6               the science of the total environment             nan  ...   \n",
       "7                   journal of environmental quality             nan  ...   \n",
       "8                                            talanta             nan  ...   \n",
       "9                                            talanta             nan  ...   \n",
       "\n",
       "                                Authors Affiliations Times cited  \\\n",
       "0  gielen, gerty j. h. p. (scion); clinton, peter...          14   \n",
       "1  song, wenlu (michigan state university); ding,...          48   \n",
       "2  wu, chenxi (university of toledo); spongberg, ...          38   \n",
       "3  monteiro, sara c. (university of york; fera sc...         133   \n",
       "4  gielen, gerty j.h.p. (scion); van den heuvel, ...          44   \n",
       "5  vazquez-roig, pablo (university of valencia); ...          74   \n",
       "6  vazquez-roig, pablo (university of valencia); ...         142   \n",
       "7  siemens, jan (university of bonn); huschek, ge...          28   \n",
       "8  chitescu, carmen lidia (\"dunarea de jos\" unive...          49   \n",
       "9  durán-alvarez, juan c. (national autonomous un...         118   \n",
       "\n",
       "   Recent citations   RCR    FCR  \\\n",
       "0                 3  0.43   1.47   \n",
       "1                 3  1.74   4.91   \n",
       "2                 4   1.2   3.97   \n",
       "3                14  4.51  13.63   \n",
       "4                 5   1.0   3.97   \n",
       "5                 9  3.08    8.9   \n",
       "6                30  5.78  16.17   \n",
       "7                 4  0.92   2.92   \n",
       "8                 7  1.83   5.77   \n",
       "9                14  3.97  11.02   \n",
       "\n",
       "                                      Source Linkout  \\\n",
       "0                                                nan   \n",
       "1                                                nan   \n",
       "2                                                nan   \n",
       "3  https://setac.onlinelibrary.wiley.com/doi/pdfd...   \n",
       "4                                                nan   \n",
       "5                                                nan   \n",
       "6                                                nan   \n",
       "7                                                nan   \n",
       "8                                                nan   \n",
       "9                                                nan   \n",
       "\n",
       "                                      Dimensions URL  \\\n",
       "0  https://app.dimensions.ai/details/publication/...   \n",
       "1  https://app.dimensions.ai/details/publication/...   \n",
       "2  https://app.dimensions.ai/details/publication/...   \n",
       "3  https://app.dimensions.ai/details/publication/...   \n",
       "4  https://app.dimensions.ai/details/publication/...   \n",
       "5  https://app.dimensions.ai/details/publication/...   \n",
       "6  https://app.dimensions.ai/details/publication/...   \n",
       "7  https://app.dimensions.ai/details/publication/...   \n",
       "8  https://app.dimensions.ai/details/publication/...   \n",
       "9  https://app.dimensions.ai/details/publication/...   \n",
       "\n",
       "                             FOR (ANZSRC) Categories  \\\n",
       "0  0605 microbiology; 0503 soil sciences; 05 envi...   \n",
       "1      0503 soil sciences; 05 environmental sciences   \n",
       "2      0503 soil sciences; 05 environmental sciences   \n",
       "3  0503 soil sciences; 05 environmental sciences;...   \n",
       "4      0503 soil sciences; 05 environmental sciences   \n",
       "5    03 chemical sciences; 0301 analytical chemistry   \n",
       "6  09 engineering; 0503 soil sciences; 03 chemica...   \n",
       "7      0503 soil sciences; 05 environmental sciences   \n",
       "8    03 chemical sciences; 0301 analytical chemistry   \n",
       "9    03 chemical sciences; 0301 analytical chemistry   \n",
       "\n",
       "  Sustainable Development Goals  \\\n",
       "0                           nan   \n",
       "1                           nan   \n",
       "2                           nan   \n",
       "3                           nan   \n",
       "4                           nan   \n",
       "5                           nan   \n",
       "6                           nan   \n",
       "7                           nan   \n",
       "8                           nan   \n",
       "9                           nan   \n",
       "\n",
       "                                     pharmaceuticals  \n",
       "0  [acetaminophen, aspirin, carbamazepine, chlorp...  \n",
       "1           [amprolium, carbadox, monensin, tylosin]  \n",
       "2  [carbamazepine, clindamycin, diltiazem, diphen...  \n",
       "3  [carbamazepine, fluoxetine, naproxen, sulfamet...  \n",
       "4                          [caffeine, carbamazepine]  \n",
       "5  [ciprofloxacin, clofibric acid, diclofenac, ib...  \n",
       "6  [acetaminophen, carbamazepine, codeine, ibupro...  \n",
       "7  [bezafibrate, clarithromycin, clindamycin, dic...  \n",
       "8                    [oxytetracycline, tetracycline]  \n",
       "9                [carbamazepine, estradiol, estrone]  \n",
       "\n",
       "[10 rows x 33 columns]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_analysis = df_analysis.reset_index(drop = True)\n",
    "df_analysis.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "96059fe0-cfc4-4b4c-8fef-9d6141cd6131",
   "metadata": {
    "id": "96059fe0-cfc4-4b4c-8fef-9d6141cd6131"
   },
   "outputs": [],
   "source": [
    "#skipped step: manual catching of the missed PPCP names on a random sample of length 20\n",
    "#random_sample = df_analysis.iloc[:,[6,32]].sample(20).reset_index(drop=True)\n",
    "\n",
    "#for i, row in random_sample.iterrows():\n",
    "#    print(row['pharmaceuticals'])\n",
    "#    print(row['Abstract'])\n",
    "#    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "940df5f8-1715-4dd8-a4a2-6adf77eaf813",
   "metadata": {
    "id": "940df5f8-1715-4dd8-a4a2-6adf77eaf813"
   },
   "source": [
    "## Analysis list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "77c51f14-dc38-451a-8cf7-b38aa1d7b916",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "77c51f14-dc38-451a-8cf7-b38aa1d7b916",
    "outputId": "0354cf0e-de20-406d-a34c-951212294df7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['acetaminophen', 'amprolium', 'aspirin', 'carbadox', 'carbamazepine', 'chlorpromazine', 'monensin', 'salicylic acid', 'tetracycline', 'tylosin']\n",
      "2732\n"
     ]
    }
   ],
   "source": [
    "#setting of a list of pharmaceuticals names occurred in the abstracts\n",
    "analysis_list = []\n",
    "for items in df_analysis['pharmaceuticals'].tolist():\n",
    "    for item in items:\n",
    "        analysis_list.append(item)\n",
    "print(sorted(analysis_list[0:10]))\n",
    "print(len(analysis_list))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24ce3f71-ad88-49e3-8c5c-f7037c5bea6a",
   "metadata": {
    "id": "24ce3f71-ad88-49e3-8c5c-f7037c5bea6a"
   },
   "source": [
    "## Occurrence analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "fd8f2005-373e-4272-979e-f9a9cfebd7d4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "fd8f2005-373e-4272-979e-f9a9cfebd7d4",
    "outputId": "c3f98aec-c95e-4544-be7b-31b42de42e81"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pharmaceutical_name</th>\n",
       "      <th>theraupeutic_group</th>\n",
       "      <th>target_group</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>paracetamol</td>\n",
       "      <td>analgesics</td>\n",
       "      <td>human</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>aspirin</td>\n",
       "      <td>nonsteroidal anti-inflammatory drugs</td>\n",
       "      <td>human and veterinary</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>carbamazepine</td>\n",
       "      <td>antiepileptic drugs</td>\n",
       "      <td>human</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>chlorpromazine</td>\n",
       "      <td>dopamine agonist</td>\n",
       "      <td>human</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>salicylic acid</td>\n",
       "      <td>skin care</td>\n",
       "      <td>human</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  pharmaceutical_name                    theraupeutic_group  \\\n",
       "0         paracetamol                            analgesics   \n",
       "1             aspirin  nonsteroidal anti-inflammatory drugs   \n",
       "2       carbamazepine                   antiepileptic drugs   \n",
       "3      chlorpromazine                      dopamine agonist   \n",
       "4      salicylic acid                             skin care   \n",
       "\n",
       "           target_group  \n",
       "0                 human  \n",
       "1  human and veterinary  \n",
       "2                 human  \n",
       "3                 human  \n",
       "4                 human  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#creating of a new dataframe containing the info with mentions of pharmaceuticals in abstracts \n",
    "#and corresponding therapeutic group\n",
    "occurrence_data = pd.DataFrame(analysis_list, columns =['pharmaceutical_name'])\n",
    "occurrence_data['theraupeutic_group'] = occurrence_data['pharmaceutical_name'].map(therapeutic_group_keys)\n",
    "occurrence_data['target_group'] = occurrence_data['pharmaceutical_name'].map(target_group_keys)\n",
    "# replace of lost item\n",
    "occurrence_data['pharmaceutical_name'].replace('acetaminophen','paracetamol', regex=True, inplace=True)\n",
    "occurrence_data['pharmaceutical_name'].replace('acetylsalicylic acid','aspirin', regex=True, inplace=True)\n",
    "\n",
    "#checking new data\n",
    "occurrence_data.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "74a6d2a8-6af8-467c-8412-fff7aa1c4400",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "74a6d2a8-6af8-467c-8412-fff7aa1c4400",
    "outputId": "27f2efc6-9afe-46b9-9bdb-c5d43c9aeeb9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "337"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#check unique names of analytes\n",
    "len(occurrence_data.pharmaceutical_name.unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bfe266b-fef2-4598-88b2-2f936bb1042a",
   "metadata": {
    "id": "9bfe266b-fef2-4598-88b2-2f936bb1042a"
   },
   "source": [
    "### Most frequently studied substances (mentions more than 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "ae1d3cfe-0c7f-4c74-98a5-710241f501b5",
   "metadata": {
    "id": "ae1d3cfe-0c7f-4c74-98a5-710241f501b5"
   },
   "outputs": [],
   "source": [
    "#setting the citation threshold\n",
    "threshold = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "666c6799-19bf-43c4-9bf0-413ad09b425b",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 162
    },
    "id": "666c6799-19bf-43c4-9bf0-413ad09b425b",
    "outputId": "b2bb1b99-a7c7-4b64-d2af-d891d1679700"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pharmaceutical_name</th>\n",
       "      <th>counts</th>\n",
       "      <th>therapeutic_group</th>\n",
       "      <th>environment</th>\n",
       "      <th>target_group</th>\n",
       "      <th>perc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>carbamazepine</td>\n",
       "      <td>207</td>\n",
       "      <td>antiepileptic drugs</td>\n",
       "      <td>soil</td>\n",
       "      <td>human</td>\n",
       "      <td>10.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ibuprofen</td>\n",
       "      <td>143</td>\n",
       "      <td>analgesics</td>\n",
       "      <td>soil</td>\n",
       "      <td>human</td>\n",
       "      <td>7.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>diclofenac</td>\n",
       "      <td>130</td>\n",
       "      <td>analgesics</td>\n",
       "      <td>soil</td>\n",
       "      <td>human</td>\n",
       "      <td>6.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  pharmaceutical_name  counts    therapeutic_group environment target_group  \\\n",
       "0       carbamazepine     207  antiepileptic drugs        soil        human   \n",
       "1           ibuprofen     143           analgesics        soil        human   \n",
       "2          diclofenac     130           analgesics        soil        human   \n",
       "\n",
       "   perc  \n",
       "0  10.3  \n",
       "1   7.1  \n",
       "2   6.5  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "51\n"
     ]
    }
   ],
   "source": [
    "#grouping of the analysis data by substances and according to the threshold\n",
    "frequently_studied_pharmaceuticals = occurrence_data['pharmaceutical_name'].value_counts(\n",
    ")[occurrence_data['pharmaceutical_name'].value_counts(\n",
    ")>threshold].rename_axis('pharmaceutical_name').reset_index(name='counts')#.hist()\n",
    "\n",
    "#seting important variables from created dictionaries\n",
    "frequently_studied_pharmaceuticals['therapeutic_group'] = frequently_studied_pharmaceuticals['pharmaceutical_name'].map(therapeutic_group_keys)\n",
    "frequently_studied_pharmaceuticals['environment'] = 'soil'\n",
    "frequently_studied_pharmaceuticals['target_group'] = frequently_studied_pharmaceuticals['pharmaceutical_name'].map(target_group_keys)\n",
    "\n",
    "#setting mentions percentage from all number of mentions in subsample\n",
    "frequently_studied_pharmaceuticals['perc']=100*frequently_studied_pharmaceuticals['counts']/frequently_studied_pharmaceuticals['counts'].sum()\n",
    "#rounding percentage variable\n",
    "frequently_studied_pharmaceuticals['perc'] = frequently_studied_pharmaceuticals['perc'].round(decimals=1)\n",
    "\n",
    "display(frequently_studied_pharmaceuticals.head(3))\n",
    "print(len(frequently_studied_pharmaceuticals))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "c3d08d93-1e35-4dcf-b24f-19bcc3a4d582",
   "metadata": {
    "id": "c3d08d93-1e35-4dcf-b24f-19bcc3a4d582"
   },
   "outputs": [],
   "source": [
    "#save results\n",
    "#output_name = 'result_tables/frequently_studied_pharmaceuticals.csv'\n",
    "#reference_df.to_csv(output_name) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "9ded9ffc-2efc-40c9-a613-0779835a0d33",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 143
    },
    "id": "9ded9ffc-2efc-40c9-a613-0779835a0d33",
    "outputId": "076b6f4e-a304-45dd-f2dd-16efce0e5ca7"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>theraupeutic_group</th>\n",
       "      <th>counts</th>\n",
       "      <th>environment</th>\n",
       "      <th>perc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>antibiotics</td>\n",
       "      <td>928</td>\n",
       "      <td>soil</td>\n",
       "      <td>36.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>analgesics</td>\n",
       "      <td>490</td>\n",
       "      <td>soil</td>\n",
       "      <td>19.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>antiepileptic drugs</td>\n",
       "      <td>214</td>\n",
       "      <td>soil</td>\n",
       "      <td>8.3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    theraupeutic_group  counts environment  perc\n",
       "0         antibiotics      928        soil  36.2\n",
       "1           analgesics     490        soil  19.1\n",
       "2  antiepileptic drugs     214        soil   8.3"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#grouping of the analysis data by the therapeutic group\n",
    "frequently_studied_theur_grops = occurrence_data['theraupeutic_group'].value_counts(\n",
    ")[occurrence_data['theraupeutic_group'].value_counts(\n",
    ")>threshold].rename_axis('theraupeutic_group').reset_index(name='counts')\n",
    "frequently_studied_theur_grops['environment'] = 'soil'\n",
    "\n",
    "#setting mentions percentage from all number of mentions in subsample\n",
    "frequently_studied_theur_grops['perc'] = 100*frequently_studied_theur_grops['counts']/frequently_studied_theur_grops['counts'].sum()\n",
    "#rounding percentage variable \n",
    "frequently_studied_theur_grops['perc'] = frequently_studied_theur_grops['perc'].round(decimals=1)\n",
    "\n",
    "frequently_studied_theur_grops.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "635894b3-65cf-49e0-b706-490c2012867c",
   "metadata": {
    "id": "635894b3-65cf-49e0-b706-490c2012867c"
   },
   "outputs": [],
   "source": [
    "#save results\n",
    "#output_name = 'result_tables/frequently_studied_theur_grops.csv'\n",
    "#reference_df.to_csv(output_name) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "828d78d5-4f38-4e78-8ed8-30dcfdd17fd6",
   "metadata": {
    "id": "828d78d5-4f38-4e78-8ed8-30dcfdd17fd6"
   },
   "source": [
    "## Results visualisation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dce6e0f9-a242-484d-a11f-04bb1354438e",
   "metadata": {
    "id": "dce6e0f9-a242-484d-a11f-04bb1354438e"
   },
   "source": [
    "### Whole timeline (from 2000 to 2022)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6435a566-5e54-4e15-9c8a-11c1a910e0f1",
   "metadata": {
    "id": "6435a566-5e54-4e15-9c8a-11c1a910e0f1",
    "tags": []
   },
   "source": [
    "#### Therapeutical groups pie plotly vis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "0373b30b-cd03-466a-a8f5-8853d55062d5",
   "metadata": {
    "id": "0373b30b-cd03-466a-a8f5-8853d55062d5"
   },
   "outputs": [],
   "source": [
    "#library for dynamic visualisations (jupyter notebook/lab, Google Colab)\n",
    "import plotly.express as px\n",
    "import importlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "21f2b383-0061-4a67-8b7e-30f181a76abc",
   "metadata": {
    "id": "21f2b383-0061-4a67-8b7e-30f181a76abc"
   },
   "outputs": [],
   "source": [
    "#defining coloring settings: pallette (pal) and number of colors (n)\n",
    "n = len(frequently_studied_theur_grops)\n",
    "pal = list(sns.color_palette(palette=\"light:#5A9\", n_colors=n).as_hex())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c93703d2-1e24-4ddb-af6f-2a1e21c746fb",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 937
    },
    "id": "c93703d2-1e24-4ddb-af6f-2a1e21c746fb",
    "outputId": "e2ff5b16-8ac7-4339-b272-2ad2d483e621"
   },
   "outputs": [],
   "source": [
    "#visualisation\n",
    "fig = px.pie(frequently_studied_theur_grops, \n",
    "             values='perc', \n",
    "             names='theraupeutic_group',\n",
    "             color_discrete_sequence=pal)\n",
    "\n",
    "fig.update_traces(textposition='outside', \n",
    "                  textinfo='percent+label', \n",
    "                  hole=.6, \n",
    "                  hoverinfo=\"label+percent+name\", \n",
    "                  textfont_size=14)\n",
    "\n",
    "fig.update_layout(width = 1600, height = 900,\n",
    "                  margin = dict(t=0, l=0, r=600, b=0),\n",
    "                  legend_font_size = 14)\n",
    "\n",
    "#output_name = 'result_plots/pie_plot.png'\n",
    "#fig.write_image(output_name, scale=6)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "591a3f54-3e3e-4d96-a6a0-bd395b4df4a6",
   "metadata": {
    "id": "591a3f54-3e3e-4d96-a6a0-bd395b4df4a6"
   },
   "source": [
    "#### Analyte names as parallel categories vis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c2b3d46-a899-4f8e-a737-5ea2dd4a27e4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 937
    },
    "id": "0c2b3d46-a899-4f8e-a737-5ea2dd4a27e4",
    "outputId": "f181039b-ea80-42aa-eea5-7602eb0df1b5"
   },
   "outputs": [],
   "source": [
    "fig = px.parallel_categories(frequently_studied_pharmaceuticals.sort_values(by=['counts',\n",
    "                                                                                'pharmaceutical_name'], \n",
    "                                                                            ascending=False), \n",
    "                             dimensions=['pharmaceutical_name','target_group','therapeutic_group'], \n",
    "                             color='counts', \n",
    "                             labels = {'pharmaceutical_name':'',\n",
    "                                       'target_group':'',\n",
    "                                       'therapeutic_group':''},\n",
    "                            height=900,\n",
    "                            width=1400)\n",
    "\n",
    "fig.update_traces(tickfont={'size': 12})\n",
    "# move colorbar\n",
    "fig.update_layout(coloraxis_colorbar=dict(orientation=\"h\"),\n",
    "                  coloraxis_colorbar_thickness=10,\n",
    "                  legend_font_size = 14)\n",
    "\n",
    "# output_name = 'result_plots/pharmaceuticals_pc_whole_timeline.png'\n",
    "# fig.write_image(output_name, scale=4)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d1058b0-c092-44be-98a7-2ae80e219877",
   "metadata": {
    "id": "7d1058b0-c092-44be-98a7-2ae80e219877"
   },
   "source": [
    "#### Analyte names as parallel categories vis -- limited occurrence (mentions perc >1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bd95d01-572a-4f28-9e55-53b3f6692884",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 637
    },
    "id": "1bd95d01-572a-4f28-9e55-53b3f6692884",
    "outputId": "a2a12168-dfe1-46bd-f763-a70618ecdcb6"
   },
   "outputs": [],
   "source": [
    "fig = px.parallel_categories(frequently_studied_pharmaceuticals.query('perc>1').sort_values(by=['counts',\n",
    "                                                                                'pharmaceutical_name'], \n",
    "                                                                            ascending=False), \n",
    "                             dimensions=['pharmaceutical_name','target_group','therapeutic_group'], \n",
    "                             color='counts', \n",
    "                             labels = {'pharmaceutical_name':'',\n",
    "                                       'target_group':'',\n",
    "                                       'therapeutic_group':''},\n",
    "                            height=600,\n",
    "                            width=1200)\n",
    "\n",
    "fig.update_traces(tickfont={'size': 12})\n",
    "# move colorbar\n",
    "fig.update_layout(coloraxis_colorbar=dict(orientation=\"h\"),\n",
    "                  coloraxis_colorbar_thickness=10)\n",
    "\n",
    "#output_name = 'result_plots/pharmaceuticals_pc_whole_timeline_1_perc.png'\n",
    "#fig.write_image(output_name, scale=4)\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28be41c3-ab4f-472e-8b7a-288cebb63698",
   "metadata": {
    "id": "28be41c3-ab4f-472e-8b7a-288cebb63698"
   },
   "source": [
    "## Last 5 years"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81d1d375-6e7a-435f-8965-81d282d3c903",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 162
    },
    "id": "81d1d375-6e7a-435f-8965-81d282d3c903",
    "outputId": "3941ba07-4d17-49df-e88c-151c2573afa1"
   },
   "outputs": [],
   "source": [
    "analysis_list_recent = []\n",
    "for items in df_analysis.query('PubYear >= 2017')['pharmaceuticals'].tolist():\n",
    "    for item in items:\n",
    "        analysis_list_recent.append(item)\n",
    "#creating of a new dataframe containing the info with mentions of pharmaceuticals in abstracts \n",
    "#and corresponding therapeutic group\n",
    "occurrence_data_recent = pd.DataFrame(analysis_list_recent, columns =['pharmaceutical_name'])\n",
    "occurrence_data_recent['theraupeutic_group'] = occurrence_data_recent['pharmaceutical_name'].map(therapeutic_group_keys)\n",
    "occurrence_data_recent['target_group'] = occurrence_data_recent['pharmaceutical_name'].map(target_group_keys)\n",
    "# replace of lost item\n",
    "occurrence_data_recent['pharmaceutical_name'].replace('acetaminophen','paracetamol', regex=True, inplace=True)\n",
    "#checking new data\n",
    "display(occurrence_data_recent.head(3))\n",
    "#check unique names of analytes\n",
    "print(len(occurrence_data_recent.pharmaceutical_name.unique()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5070fd94-3569-405f-93a0-f1c8d3af9d82",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5070fd94-3569-405f-93a0-f1c8d3af9d82",
    "outputId": "019ae31a-0b91-414e-a008-e88965cc60c2"
   },
   "outputs": [],
   "source": [
    "#grouping of the analysis data by substances\n",
    "frequently_studied_pharmaceuticals_r = occurrence_data_recent['pharmaceutical_name'].value_counts()[occurrence_data_recent['pharmaceutical_name'].value_counts(\n",
    ")>threshold].rename_axis('pharmaceutical_name').reset_index(name='counts')#.hist()\n",
    "#seting important variables\n",
    "frequently_studied_pharmaceuticals_r['therapeutic_group'] = frequently_studied_pharmaceuticals_r['pharmaceutical_name'].map(therapeutic_group_keys)\n",
    "frequently_studied_pharmaceuticals_r['environment'] = 'soil'\n",
    "frequently_studied_pharmaceuticals_r['target_group'] = frequently_studied_pharmaceuticals_r['pharmaceutical_name'].map(target_group_keys)\n",
    "print(len(frequently_studied_pharmaceuticals_r))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83d7ed0f-a148-4f9b-aab0-63fc67056141",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 737
    },
    "id": "83d7ed0f-a148-4f9b-aab0-63fc67056141",
    "outputId": "32c8329d-7360-43db-c6b7-3020b5369f4b"
   },
   "outputs": [],
   "source": [
    "fig = px.parallel_categories(frequently_studied_pharmaceuticals_r.sort_values(by=['counts',\n",
    "                                                                                'pharmaceutical_name'], \n",
    "                                                                            ascending=False), \n",
    "                             dimensions=['pharmaceutical_name','target_group','therapeutic_group'], \n",
    "                             color='counts',\n",
    "                             labels = {'pharmaceutical_name':'',\n",
    "                                       'target_group':'',\n",
    "                                       'therapeutic_group':''},\n",
    "                            height=700,\n",
    "\n",
    "                             width=1200).update_traces(labelfont={\"size\":18})\n",
    "\n",
    "# move colorbar\n",
    "fig.update_layout(coloraxis_colorbar=dict(orientation=\"h\"),\n",
    "                  coloraxis_colorbar_thickness=10)\n",
    "\n",
    "#output_name = 'result_plots/pharmaceuticals_pc_from_2017.png'\n",
    "#fig.write_image(output_name, scale=6)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22c19f9d-6c54-4873-b480-4f57795ba9d9",
   "metadata": {
    "id": "22c19f9d-6c54-4873-b480-4f57795ba9d9"
   },
   "outputs": [],
   "source": [
    "#grouping of the analysis data by the therapeutic group\n",
    "frequently_studied_theur_grops_r = occurrence_data_recent['theraupeutic_group'].value_counts(\n",
    ")[occurrence_data_recent['theraupeutic_group'].value_counts(\n",
    ")>threshold].rename_axis('theraupeutic_group').reset_index(name='counts')\n",
    "frequently_studied_theur_grops_r['perc'] = 100*frequently_studied_theur_grops_r['counts']/frequently_studied_theur_grops_r['counts'].sum()\n",
    "#\n",
    "frequently_studied_theur_grops_r['perc'] =frequently_studied_theur_grops_r['perc'].round(decimals=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "705fa1e3-8454-4d4c-ad9b-b876f6c0d3b8",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 887
    },
    "id": "705fa1e3-8454-4d4c-ad9b-b876f6c0d3b8",
    "outputId": "2141a21f-62e9-4611-b710-db0be3ad7289"
   },
   "outputs": [],
   "source": [
    "fig = px.pie(frequently_studied_theur_grops_r, \n",
    "             values='perc', \n",
    "             names='theraupeutic_group',\n",
    "             color_discrete_sequence=pal)\n",
    "\n",
    "fig.update_traces(textposition='outside', textinfo='percent+label', \n",
    "                  hole=.6, hoverinfo=\"label+percent+name\")\n",
    "\n",
    "fig.update_layout(width = 1200, height = 850,\n",
    "                  margin = dict(t=0, l=0, r=0, b=0))\n",
    "\n",
    "#output_name = 'result_plots/pie_plot_from_2017.png'\n",
    "#fig.write_image(output_name, scale=6)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bec754e-7434-4b0c-939e-f0ecb735db10",
   "metadata": {
    "id": "0bec754e-7434-4b0c-939e-f0ecb735db10"
   },
   "source": [
    "## Before 2017"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0d986fd-bdf6-4e86-b99a-12afbd8cc062",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 225
    },
    "id": "d0d986fd-bdf6-4e86-b99a-12afbd8cc062",
    "outputId": "6f7a65f9-09e9-43dc-a7ed-e4569181e539"
   },
   "outputs": [],
   "source": [
    "df_before_2017 = df_analysis.query('PubYear < 2017')\n",
    "analysis_list_before_2017 = []\n",
    "\n",
    "for items in df_before_2017['pharmaceuticals'].tolist():\n",
    "    for item in items:\n",
    "        analysis_list_before_2017.append(item)\n",
    "\n",
    "#creating of a new dataframe containing the info with mentions of pharmaceuticals in abstracts \n",
    "#and corresponding therapeutic group\n",
    "occurrence_data_before_2017 = pd.DataFrame(analysis_list_before_2017, columns =['pharmaceutical_name'])\n",
    "occurrence_data_before_2017['theraupeutic_group'] = occurrence_data_before_2017['pharmaceutical_name'].map(therapeutic_group_keys)\n",
    "occurrence_data_before_2017['target_group'] = occurrence_data_before_2017['pharmaceutical_name'].map(target_group_keys)\n",
    "# replace of lost item\n",
    "occurrence_data_before_2017['pharmaceutical_name'].replace('acetaminophen','paracetamol', regex=True, inplace=True)\n",
    "#checking new data\n",
    "display(occurrence_data_before_2017.head(5))\n",
    "#check unique names of analytes\n",
    "print(len(occurrence_data_before_2017.pharmaceutical_name.unique()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "800744f0-4b26-4f77-89c3-27b8c7a43f11",
   "metadata": {
    "id": "800744f0-4b26-4f77-89c3-27b8c7a43f11"
   },
   "outputs": [],
   "source": [
    "#grouping of the analysis data by substances\n",
    "frequently_studied_pharmaceuticals_b = occurrence_data_before_2017['pharmaceutical_name'].value_counts(\n",
    ")[occurrence_data_before_2017['pharmaceutical_name'].value_counts()>threshold].rename_axis('pharmaceutical_name').reset_index(name='counts')#.hist()\n",
    "\n",
    "#seting important variables\n",
    "frequently_studied_pharmaceuticals_b['therapeutic_group'] = frequently_studied_pharmaceuticals_b['pharmaceutical_name'].map(therapeutic_group_keys)\n",
    "frequently_studied_pharmaceuticals_b['environment'] = 'soil'\n",
    "frequently_studied_pharmaceuticals_b['target_group'] = frequently_studied_pharmaceuticals_b['pharmaceutical_name'].map(target_group_keys)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58ab6eba-eeaf-421d-92a3-82968505835e",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 737
    },
    "id": "58ab6eba-eeaf-421d-92a3-82968505835e",
    "outputId": "2a989f4b-89cf-4a11-be11-5587ee742363"
   },
   "outputs": [],
   "source": [
    "fig = px.parallel_categories(frequently_studied_pharmaceuticals_b.sort_values(by=['counts',\n",
    "                                                                                'pharmaceutical_name'], \n",
    "                                                                            ascending=False), \n",
    "                             dimensions=['pharmaceutical_name','target_group','therapeutic_group'], \n",
    "                             color='counts',\n",
    "                             labels = {'pharmaceutical_name':'',\n",
    "                                       'target_group':'',\n",
    "                                       'therapeutic_group':''},\n",
    "                            height=700,\n",
    "                            width=1200).update_traces(labelfont={\"size\":18})\n",
    "\n",
    "# move colorbar\n",
    "fig.update_layout(coloraxis_colorbar=dict(orientation=\"h\"),\n",
    "                  coloraxis_colorbar_thickness=10)\n",
    "\n",
    "#output_name = 'result_plots/pharmaceuticals_pc_before_2017.png'\n",
    "#fig.write_image(output_name, scale=6)\n",
    "\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e5f349a-1f72-4e9f-8f1e-bd9bdb872253",
   "metadata": {
    "id": "6e5f349a-1f72-4e9f-8f1e-bd9bdb872253"
   },
   "outputs": [],
   "source": [
    "#grouping of the analysis data by the therapeutic group\n",
    "frequently_studied_pharmaceuticals_b = occurrence_data_before_2017['theraupeutic_group'].value_counts(\n",
    ")[occurrence_data_before_2017['theraupeutic_group'].value_counts(\n",
    ")>threshold].rename_axis('theraupeutic_group').reset_index(name='counts')\n",
    "\n",
    "frequently_studied_pharmaceuticals_b['perc'] = 100*frequently_studied_pharmaceuticals_b['counts']/frequently_studied_pharmaceuticals_b['counts'].sum()\n",
    "#\n",
    "frequently_studied_pharmaceuticals_b['perc'] = frequently_studied_pharmaceuticals_b['perc'].round(decimals=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18880c01-9734-4a8d-a3c0-f92cb6d51395",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 887
    },
    "id": "18880c01-9734-4a8d-a3c0-f92cb6d51395",
    "outputId": "ddeac64a-b924-4940-a070-0908efa04d89"
   },
   "outputs": [],
   "source": [
    "fig = px.pie(frequently_studied_pharmaceuticals_b, \n",
    "             values='perc', \n",
    "             names='theraupeutic_group',\n",
    "             color_discrete_sequence=pal)\n",
    "\n",
    "fig.update_traces(textposition='outside', textinfo='percent+label', \n",
    "                  hole=.6, hoverinfo=\"label+percent+name\")\n",
    "\n",
    "fig.update_layout(width = 1200, height = 850,\n",
    "                  margin = dict(t=0, l=0, r=0, b=0))\n",
    "\n",
    "#output_name = 'result_plots/pie_plot_before_2017.png'\n",
    "#fig.write_image(output_name, scale=6)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "304374a8-4b40-4343-be15-0e03354b6d88",
   "metadata": {
    "id": "304374a8-4b40-4343-be15-0e03354b6d88"
   },
   "source": [
    "## Top cited papers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3deb7071-3fe1-410b-b908-9121a21440d5",
   "metadata": {
    "id": "3deb7071-3fe1-410b-b908-9121a21440d5"
   },
   "source": [
    "On this stage we select the most cited papers by pharmaceuticals and among the last 5 years"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4409a61-e861-4273-8d34-fa28b6d823b2",
   "metadata": {
    "id": "d4409a61-e861-4273-8d34-fa28b6d823b2"
   },
   "outputs": [],
   "source": [
    "#top cited papers by mention of analytes\n",
    "df_analysis = df_analysis.apply(lambda x: x.astype(str).str.lower(\n",
    ") if x.name != 'Times cited' and x.name != 'PubYear' else x)\n",
    "reference_df = pd.DataFrame()\n",
    "\n",
    "for pharm in frequently_studied_pharmaceuticals.pharmaceutical_name:\n",
    "    reference_data = df_analysis[df_analysis.pharmaceuticals.str.contains(pharm)][['DOI', \n",
    "                                                        'Title',\n",
    "                                                        'Source title',\n",
    "                                                        'PubYear',\n",
    "                                                        'Times cited',\n",
    "                                                        'pharmaceuticals']].query('PubYear >= 2017').sort_values(by='Times cited',\n",
    "                                                                                  ascending=False).head(5)\n",
    "    reference_data['pharmaceutical_name'] = pharm\n",
    "    reference_df = pd.concat((reference_df,reference_data)).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c6148ec-f04b-4f48-8173-0ad84a4df84a",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 676
    },
    "id": "2c6148ec-f04b-4f48-8173-0ad84a4df84a",
    "outputId": "1ca69a1d-d3a5-4174-b17c-0a8bc58c9ebd"
   },
   "outputs": [],
   "source": [
    "reference_df#.head(10)\n",
    "\n",
    "#save results\n",
    "output_name = 'result_tables/reference_df.csv'\n",
    "reference_df.to_csv(output_name) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d92b4afa-5fcc-482a-8460-1a3d5f957a05",
   "metadata": {
    "id": "d92b4afa-5fcc-482a-8460-1a3d5f957a05"
   },
   "outputs": [],
   "source": [
    "reference_df['reference_number'] = reference_df.groupby('Title', sort=False).ngroup()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "174b50c3-d12e-431f-a3de-8d97fb6c8ae2",
   "metadata": {
    "id": "174b50c3-d12e-431f-a3de-8d97fb6c8ae2"
   },
   "outputs": [],
   "source": [
    "# display table with total amount of papers with mentions from 2000 to 2022\n",
    "#pd.merge(reference_df, frequently_studied_pharmaceuticals[['pharmaceutical_name','counts']],on='pharmaceutical_name', how='left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b4bc20e-47c7-41fb-a982-6aec754222e8",
   "metadata": {
    "id": "6b4bc20e-47c7-41fb-a982-6aec754222e8"
   },
   "outputs": [],
   "source": [
    "#searching through each year top-5 cited publicaitons\n",
    "#for year in df_analysis['PubYear'].unique():\n",
    "#    display(df_analysis[df_analysis['PubYear'] == year][['DOI', \n",
    "#                                                        'Title',\n",
    "#                                                        'Source title',\n",
    "#                                                        'PubYear',\n",
    "#                                                        'Times cited',\n",
    "#                                                        'pharmaceuticals']].sort_values(by='Times cited', \n",
    "#                                                                    ascending=False).head(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92df50bc-1c2e-4e06-b8ff-ce6a289b8256",
   "metadata": {
    "id": "92df50bc-1c2e-4e06-b8ff-ce6a289b8256"
   },
   "source": [
    "# 🍂"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "provenance": []
  },
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "venv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}