map-and-score.ipynb

{
    "cells": [
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "# Disruptive Papers in Fetal Surgery"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Description\n",
                "\n",
                "This notebook takes as input one or more PubMed CSV exports, placed in the [data/pubmed/results/](data/pubmed/results/) directory. It adds useful columns to the CSV file, maps the PubMed IDs to Microsoft Academic Graph IDs, uses the MAGIDs to look up citation and disruption data from Wu et al., 2023 to score each article, and saves the scored output to a styled Excel spreadsheet.\n",
                "\n",
                "### PubMed Search Documentation and Replication\n",
                "\n",
                "The searches used to create the PubMed CSV exports are documented in [DisruptivePapersFetalSurgery-PubMedSearchDocumentation-202303.xlsx](data/pubmed/searches/DisruptivePapersFetalSurgery-PubMedSearchDocumentation-202303.xlsx). They cover Congenital Diaphragmatic Hernia (CDH), Congenital Pulmonary Airway Malfunction (CPAM), Neural Tube Defects (NTD), and Twin-to-Twin Transfusion Syndrome (TTTS), with each search documented on a separate spreadsheet tab. \n",
                "\n",
                "The searches can be replicated by either copying and pasting the full search string from each spreadsheet tab into PubMed or clicking on the number in the Results column. Once replicated, the search results can be exported to CSV via the *Save* button. Select *All results* and *CSV* format, then click *Create file*.\n",
                "\n",
                "### References\n",
                "\n",
                "Wu, L., Wang, D., & Evans, J. (2023). Replication Data for: Large teams develop and small teams disrupt science and technology [Data set]. Harvard Dataverse. https://doi.org/10.7910/DVN/JPWNNK\n"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Environment"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### Import libraries"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 33,
            "metadata": {},
            "outputs": [],
            "source": [
                "import glob\n",
                "import logging\n",
                "import pandas as pd\n",
                "from pathlib import Path\n",
                "import requests\n",
                "from styleframe import StyleFrame, Styler, utils\n",
                "import time\n",
                "from tqdm.notebook import tqdm\n"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### Define constants"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 34,
            "metadata": {},
            "outputs": [],
            "source": [
                "email = \"whimar@ohsu.edu\"\n",
                "openalex = \"https://api.openalex.org/works?per-page=100&filter=pmid:\"\n",
                "headers = {\"user-agent\": \"mailto:\" + email}\n",
                "\n",
                "datestamp = time.strftime(\"%Y%m%d\")\n",
                "TOP_PAPERS = 100\n",
                "\n",
                "DATA_DIR = \"data/\"\n",
                "DISRUPT_DIR = DATA_DIR + \"MAG-disruption/\"\n",
                "PUBMED_DIR = DATA_DIR + \"pubmed/\"\n",
                "RESULTS_DIR = PUBMED_DIR + \"results/*\"\n",
                "\n",
                "LOG_DIR = \"log/\"\n",
                "\n",
                "TARGET_DIR = \"target/\"\n",
                "CSV_DIR = TARGET_DIR + \"csv/\"\n",
                "XLSX_DIR = TARGET_DIR + \"xlsx/\"\n"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### Create target directories"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 35,
            "metadata": {},
            "outputs": [],
            "source": [
                "Path(LOG_DIR).mkdir(parents=True, exist_ok=True)\n",
                "Path(TARGET_DIR).mkdir(parents=True, exist_ok=True)\n",
                "Path(CSV_DIR).mkdir(parents=True, exist_ok=True)\n",
                "Path(XLSX_DIR).mkdir(parents=True, exist_ok=True)\n"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### Configure logging"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 36,
            "metadata": {},
            "outputs": [],
            "source": [
                "logging.basicConfig(\n",
                "    filename=f\"{LOG_DIR}fsdp-{datestamp}.log\",\n",
                "    filemode=\"w\",\n",
                "    force=True,\n",
                "    format=\"%(asctime)s.%(msecs)03d : %(levelname)s : %(message)s\",\n",
                "    level=logging.INFO,\n",
                "    datefmt=\"%Y-%m-%d %H:%M:%S\",\n",
                ")\n",
                "logger = logging.getLogger()"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Function definitions"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### pmid2magid\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "\n",
                "#### Description\n",
                "\n",
                "Maps PubMed IDs (PMID) from a PubMed CSV export to Microsoft Academic Graph IDs (MAGID) using the [OpenAlex Works API](https://docs.openalex.org/api-entities/works). Caches API results to CSV and reads from CSV if it already exists to reduce API calls.\n",
                "\n",
                "#### Parameters\n",
                "\n",
                "| Parameter | Description |\n",
                "| --- | --- |\n",
                "| articles_df | A processed/augmented DataFrame created from a PubMed CSV export |\n",
                "| topicdate | The stem of the PubMed CSV filename, expected filename convenion *topic-date.csv* |\n",
                "\n",
                "#### Return Values\n",
                "\n",
                "| Return Value | Description |\n",
                "| --- | --- |\n",
                "| mags_df | a two-column dataframe of PMIDs and MAGIDs |\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 37,
            "metadata": {},
            "outputs": [],
            "source": [
                "def pmid2magid(articles_df, topicdate):\n",
                "    magsfile = f\"{CSV_DIR}{topicdate}-MAGID.csv\"\n",
                "    magsfilepath = Path(magsfile)\n",
                "\n",
                "    if magsfilepath.is_file():\n",
                "            logger.info(\"Reading cached %s MAGID data from file %s\", topicdate, magsfile)\n",
                "            mags_df = pd.read_csv(magsfile, index_col=\"long_pmid\")\n",
                "    else:\n",
                "        logger.info(\"Pulling new %s MAGID data from OpenAlex\", topicdate)\n",
                "        pmids = articles_df[\"pmid\"].tolist()\n",
                "\n",
                "        # combine PMIDs into groups of 50 to reduce OpenAlex API calls\n",
                "        pmid_groups = [pmids[i: i+50] for i in range(0, len(pmids), 50)]\n",
                "\n",
                "        pmid_strings = []\n",
                "        mags = {}\n",
                "        for pmid_group in pmid_groups:\n",
                "            pmid_strings.append(\"|\".join(map(str, pmid_group)))\n",
                "        \n",
                "        for pmid_string in tqdm(pmid_strings, desc=topicdate + \" openalex\"):\n",
                "            logger.info(\"Requesting %s PMIDs from OpenAlex: %s\", topicdate, pmid_string)\n",
                "            response = requests.get(openalex + pmid_string, headers=headers)\n",
                "            logger.info(\"OpenAlex response code: %s\", response.status_code)\n",
                "            if response.status_code == 200:\n",
                "                works = response.json()\n",
                "                for work in works[\"results\"]:\n",
                "                    if \"mag\" in work[\"ids\"]:\n",
                "                        logger.info(\"Found MAGID %s for PMID %s\", work[\"ids\"][\"mag\"], work[\"ids\"][\"pmid\"])\n",
                "                        mags[work[\"ids\"][\"pmid\"]] = work[\"ids\"][\"mag\"]\n",
                "                    else:\n",
                "                        logger.info(\"No MAGID found for PMID %s\", work[\"ids\"][\"pmid\"])\n",
                "            else:\n",
                "                logger.error(\"Error making OpenAlex request\")\n",
                "        \n",
                "        mags_df = pd.DataFrame.from_dict(mags, orient=\"index\", columns=[\"magid\"])\n",
                "        mags_df.index.name = \"long_pmid\"\n",
                "        mags_df.sort_index(inplace=True)\n",
                "        logger.info(\"Saving mapped PMID/MAGIDs to %s\", magsfile)\n",
                "        mags_df.to_csv(magsfile)\n",
                "        mags_df = pd.read_csv(magsfile, index_col=\"long_pmid\")\n",
                "\n",
                "    return mags_df"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### score_results\n"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### Description\n",
                "\n",
                "Maps PubMed IDs (PMID) from a PubMed CSV export to citation and disruption score data.\n",
                "\n",
                "#### Parameters\n",
                "\n",
                "| Parameter | Description |\n",
                "| --- | --- |\n",
                "| articles_df | A processed/augmented DataFrame created from a PubMed CSV export |\n",
                "| topicdate | The stem of the PubMed CSV filename, expected format topic-date.csv |\n",
                "\n",
                "#### Return Values\n",
                "\n",
                "| Return Value | Description |\n",
                "| --- | --- |\n",
                "| cited_df | A processed/augmented DataFrame created from a PubMed CSV export and sorted by descending number of citations |\n",
                "| develop_df | A processed/augmented DataFrame created from a PubMed CSV export and sorted from most developmental to least developmental |\n",
                "| disrupt_df |A processed/augmented DataFrame created from a PubMed CSV export and sorted from most disruptive to least disruptive |\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 38,
            "metadata": {},
            "outputs": [],
            "source": [
                "def score_results(articles_df, topicdate):\n",
                "    citefile = f\"{CSV_DIR}{topicdate}-topcited.csv\"\n",
                "    developfile = f\"{CSV_DIR}{topicdate}-topdevelopmental.csv\"\n",
                "    disruptfile = f\"{CSV_DIR}{topicdate}-topdisruptive.csv\"\n",
                "\n",
                "    # magdata_df = pd.read_csv(\n",
                "    #     f\"AggregatedMAG.txt\",\n",
                "    #     sep=\"\\t\",\n",
                "    #     usecols=[0, 5, 6],\n",
                "    #     names=[\"magid\", \"num_citations\", \"disruption_score\"],\n",
                "    #     index_col=\"magid\",\n",
                "    #     dtype={\"num_citations\": \"int64\", \"disruption_score\": \"float64\"},\n",
                "    # )\n",
                "    magdata_df = pd.read_csv(\n",
                "        f\"{DISRUPT_DIR}Aggregated_20210521.txt\",\n",
                "        sep=\"\\t\",\n",
                "        usecols=[0, 4, 5],\n",
                "        names=[\"magid\", \"num_citations\", \"disruption_score\"],\n",
                "        index_col=\"magid\",\n",
                "        dtype={\"num_citations\": \"int64\", \"disruption_score\": \"float64\"},\n",
                "    )\n",
                "    magdata_df.sort_index(inplace=True)\n",
                "\n",
                "    scored_df = articles_df.join(magdata_df)\n",
                "    scored_df.reset_index(inplace=True)\n",
                "\n",
                "    cited_df = scored_df[\n",
                "        [\n",
                "            \"num_citations\",\n",
                "            \"title\",\n",
                "            \"journal\",\n",
                "            \"pubdate\",\n",
                "            \"magid\",\n",
                "            \"pmid\",\n",
                "            \"doi\",\n",
                "            \"pubmed\",\n",
                "            \"ohsu_library\",\n",
                "            \"rush_library\",\n",
                "        ]\n",
                "    ].nlargest(TOP_PAPERS, columns=\"num_citations\")\n",
                "    development_df = scored_df[\n",
                "        [\n",
                "            \"disruption_score\",\n",
                "            \"title\",\n",
                "            \"journal\",\n",
                "            \"pubdate\",\n",
                "            \"magid\",\n",
                "            \"pmid\",\n",
                "            \"doi\",\n",
                "            \"pubmed\",\n",
                "            \"ohsu_library\",\n",
                "            \"rush_library\",\n",
                "        ]\n",
                "    ].nsmallest(TOP_PAPERS, columns=\"disruption_score\")\n",
                "    disrupt_sf = scored_df[\n",
                "        [\n",
                "            \"disruption_score\",\n",
                "            \"title\",\n",
                "            \"journal\",\n",
                "            \"pubdate\",\n",
                "            \"magid\",\n",
                "            \"pmid\",\n",
                "            \"doi\",\n",
                "            \"pubmed\",\n",
                "            \"ohsu_library\",\n",
                "            \"rush_library\",\n",
                "        ]\n",
                "    ].nlargest(TOP_PAPERS, columns=\"disruption_score\")\n",
                "\n",
                "    cited_df.to_csv(\n",
                "        citefile,\n",
                "        index=False,\n",
                "        columns=[\n",
                "            \"num_citations\",\n",
                "            \"magid\",\n",
                "            \"pmid\",\n",
                "            \"title\",\n",
                "            \"journal\",\n",
                "            \"pubdate\",\n",
                "            \"doi\",\n",
                "        ],\n",
                "    )\n",
                "    development_df.to_csv(\n",
                "        developfile,\n",
                "        index=False,\n",
                "        columns=[\n",
                "            \"disruption_score\",\n",
                "            \"magid\",\n",
                "            \"pmid\",\n",
                "            \"title\",\n",
                "            \"journal\",\n",
                "            \"pubdate\",\n",
                "            \"doi\",\n",
                "        ],\n",
                "    )\n",
                "    disrupt_sf.to_csv(\n",
                "        disruptfile,\n",
                "        index=False,\n",
                "        columns=[\n",
                "            \"disruption_score\",\n",
                "            \"magid\",\n",
                "            \"pmid\",\n",
                "            \"title\",\n",
                "            \"journal\",\n",
                "            \"pubdate\",\n",
                "            \"doi\",\n",
                "        ],\n",
                "    )\n",
                "\n",
                "    return cited_df, development_df, disrupt_sf"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### style_output\n"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### Description\n",
                "\n",
                "Exports sorted scored dataframes to styled XLSX file.\n",
                "\n",
                "#### Parameters\n",
                "\n",
                "| Parameter | Description |\n",
                "| --- | --- |\n",
                "| cited_df | A processed/augmented DataFrame created from a PubMed CSV export and sorted by descending number of citations |\n",
                "| develop_df | A processed/augmented DataFrame created from a PubMed CSV export and sorted from most developmental to least developmental |\n",
                "| disrupt_df |A processed/augmented DataFrame created from a PubMed CSV export and sorted from most disruptive to least disruptive |\n",
                "| topicdate | The stem of the PubMed CSV filename, expected format topic-date.csv |\n",
                "\n",
                "#### Return Values\n",
                "\n",
                "None."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 39,
            "metadata": {},
            "outputs": [],
            "source": [
                "def style_output(cited_df, develop_df, disrupt_sf, topicdate):\n",
                "    scorefile = f\"{XLSX_DIR}{topicdate}-scores.xlsx\"\n",
                "\n",
                "    cited_sf = StyleFrame(cited_df.drop(\"pmid\", axis=1))\n",
                "    develop_sf = StyleFrame(develop_df.drop(\"pmid\", axis=1))\n",
                "    disrupt_sf = StyleFrame(disrupt_sf.drop(\"pmid\", axis=1))\n",
                "    # query_s = pd.Series({f\"{querytype} query\": query})\n",
                "\n",
                "    default_style = Styler(\n",
                "        font=utils.fonts.calibri,\n",
                "        font_size=11,\n",
                "        border_type=utils.borders.default_grid,\n",
                "        horizontal_alignment=utils.horizontal_alignments.left,\n",
                "        wrap_text=False,\n",
                "        shrink_to_fit=False,\n",
                "    )\n",
                "\n",
                "    header_style = Styler(\n",
                "        bg_color=utils.colors.black,\n",
                "        bold=True,\n",
                "        font=utils.fonts.calibri,\n",
                "        font_color=utils.colors.white,\n",
                "        font_size=14,\n",
                "        horizontal_alignment=utils.horizontal_alignments.left,\n",
                "        shrink_to_fit=False,\n",
                "        wrap_text=False,\n",
                "        vertical_alignment=utils.vertical_alignments.center,\n",
                "    )\n",
                "    hyperlink_style = Styler(\n",
                "        font_color=utils.colors.blue,\n",
                "        protection=True,\n",
                "        underline=utils.underline.single,\n",
                "    )\n",
                "    float_style = Styler(\n",
                "        number_format=\"0.000000000000\",\n",
                "        horizontal_alignment=utils.horizontal_alignments.right,\n",
                "    )\n",
                "\n",
                "    cited_sf.set_column_width_dict(\n",
                "        col_width_dict={\n",
                "            (\"pubdate\", \"magid\", \"pubmed\"): 12,\n",
                "            (\"num_citations\", \"ohsu_library\", \"rush_library\"): 15,\n",
                "            (\"journal\"): 30,\n",
                "            (\"title\", \"doi\"): 50,\n",
                "        }\n",
                "    )\n",
                "    cited_sf.apply_headers_style(header_style)\n",
                "    cited_sf.apply_column_style(cited_sf.columns, styler_obj=default_style)\n",
                "    cited_sf.apply_column_style(\n",
                "        [\"doi\", \"pubmed\", \"ohsu_library\", \"rush_library\"],\n",
                "        styler_obj=Styler.combine(default_style, hyperlink_style),\n",
                "    )\n",
                "\n",
                "    develop_sf.set_column_width_dict(\n",
                "        col_width_dict={\n",
                "            (\"pubdate\", \"magid\", \"pubmed\"): 12,\n",
                "            (\"ohsu_library\", \"rush_library\"): 15,\n",
                "            (\"disruption_score\"): 20,\n",
                "            (\"journal\"): 30,\n",
                "            (\"title\", \"doi\"): 50,\n",
                "        }\n",
                "    )\n",
                "    develop_sf.apply_headers_style(header_style)\n",
                "    develop_sf.apply_column_style(develop_sf.columns, styler_obj=default_style)\n",
                "    develop_sf.apply_column_style(\n",
                "        \"disruption_score\", styler_obj=Styler.combine(default_style, float_style)\n",
                "    )\n",
                "    develop_sf.apply_column_style(\n",
                "        [\"doi\", \"pubmed\", \"ohsu_library\", \"rush_library\"],\n",
                "        styler_obj=Styler.combine(default_style, hyperlink_style),\n",
                "    )\n",
                "\n",
                "    disrupt_sf.set_column_width_dict(\n",
                "        col_width_dict={\n",
                "            (\"pubdate\", \"magid\", \"pubmed\"): 12,\n",
                "            (\"ohsu_library\", \"rush_library\"): 15,\n",
                "            (\"disruption_score\"): 20,\n",
                "            (\"journal\"): 30,\n",
                "            (\"title\", \"doi\"): 50,\n",
                "        }\n",
                "    )\n",
                "    disrupt_sf.apply_headers_style(header_style)\n",
                "    disrupt_sf.apply_column_style(disrupt_sf.columns, styler_obj=default_style)\n",
                "    disrupt_sf.apply_column_style(\n",
                "        \"disruption_score\", styler_obj=Styler.combine(default_style, float_style)\n",
                "    )\n",
                "    disrupt_sf.apply_column_style(\n",
                "        [\"doi\", \"pubmed\", \"ohsu_library\", \"rush_library\"],\n",
                "        styler_obj=Styler.combine(default_style, hyperlink_style),\n",
                "    )\n",
                "\n",
                "    with pd.ExcelWriter(scorefile) as sfile:\n",
                "        cited_sf.to_excel(\n",
                "            sfile,\n",
                "            index=False,\n",
                "            columns=[\n",
                "                \"num_citations\",\n",
                "                \"title\",\n",
                "                \"journal\",\n",
                "                \"pubdate\",\n",
                "                \"magid\",\n",
                "                \"doi\",\n",
                "                \"pubmed\",\n",
                "                \"ohsu_library\",\n",
                "                \"rush_library\",\n",
                "            ],\n",
                "            sheet_name=f\"top {TOP_PAPERS} cited\",\n",
                "        )\n",
                "        develop_sf.to_excel(\n",
                "            sfile,\n",
                "            index=False,\n",
                "            columns=[\n",
                "                \"disruption_score\",\n",
                "                \"title\",\n",
                "                \"journal\",\n",
                "                \"pubdate\",\n",
                "                \"magid\",\n",
                "                \"doi\",\n",
                "                \"pubmed\",\n",
                "                \"ohsu_library\",\n",
                "                \"rush_library\",\n",
                "            ],\n",
                "            sheet_name=f\"top {TOP_PAPERS} developmental\",\n",
                "        )\n",
                "        disrupt_sf.to_excel(\n",
                "            sfile,\n",
                "            index=False,\n",
                "            columns=[\n",
                "                \"disruption_score\",\n",
                "                \"title\",\n",
                "                \"journal\",\n",
                "                \"pubdate\",\n",
                "                \"magid\",\n",
                "                \"doi\",\n",
                "                \"pubmed\",\n",
                "                \"ohsu_library\",\n",
                "                \"rush_library\",\n",
                "            ],\n",
                "            sheet_name=f\"top {TOP_PAPERS} disruptive\",\n",
                "        )\n",
                "        # query_s.to_excel(sfile, index=False, header=False, sheet_name=\"pubmed query\")"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "# Read and Augment PubMed CSV for MAGID mapping and citation/disruption scoring"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 40,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "application/json": {
                            "ascii": false,
                            "bar_format": null,
                            "colour": null,
                            "elapsed": 0.008211851119995117,
                            "initial": 0,
                            "n": 0,
                            "ncols": null,
                            "nrows": null,
                            "postfix": null,
                            "prefix": "CDH-202303 openalex",
                            "rate": null,
                            "total": 6,
                            "unit": "it",
                            "unit_divisor": 1000,
                            "unit_scale": false
                        },
                        "application/vnd.jupyter.widget-view+json": {
                            "model_id": "cc9e36a2f587492c9816e0df0e0d99d2",
                            "version_major": 2,
                            "version_minor": 0
                        },
                        "text/plain": [
                            "CDH-202303 openalex:   0%|          | 0/6 [00:00<?, ?it/s]"
                        ]
                    },
                    "metadata": {},
                    "output_type": "display_data"
                },
                {
                    "data": {
                        "application/json": {
                            "ascii": false,
                            "bar_format": null,
                            "colour": null,
                            "elapsed": 0.007385730743408203,
                            "initial": 0,
                            "n": 0,
                            "ncols": null,
                            "nrows": null,
                            "postfix": null,
                            "prefix": "CPAM-202303 openalex",
                            "rate": null,
                            "total": 12,
                            "unit": "it",
                            "unit_divisor": 1000,
                            "unit_scale": false
                        },
                        "application/vnd.jupyter.widget-view+json": {
                            "model_id": "4ec0d79d3bb64c40ba27395f74c0f572",
                            "version_major": 2,
                            "version_minor": 0
                        },
                        "text/plain": [
                            "CPAM-202303 openalex:   0%|          | 0/12 [00:00<?, ?it/s]"
                        ]
                    },
                    "metadata": {},
                    "output_type": "display_data"
                },
                {
                    "data": {
                        "application/json": {
                            "ascii": false,
                            "bar_format": null,
                            "colour": null,
                            "elapsed": 0.007043123245239258,
                            "initial": 0,
                            "n": 0,
                            "ncols": null,
                            "nrows": null,
                            "postfix": null,
                            "prefix": "NTD-202303 openalex",
                            "rate": null,
                            "total": 66,
                            "unit": "it",
                            "unit_divisor": 1000,
                            "unit_scale": false
                        },
                        "application/vnd.jupyter.widget-view+json": {
                            "model_id": "8806c60ed4ad47cc89ca04b8ec0b74af",
                            "version_major": 2,
                            "version_minor": 0
                        },
                        "text/plain": [
                            "NTD-202303 openalex:   0%|          | 0/66 [00:00<?, ?it/s]"
                        ]
                    },
                    "metadata": {},
                    "output_type": "display_data"
                },
                {
                    "data": {
                        "application/json": {
                            "ascii": false,
                            "bar_format": null,
                            "colour": null,
                            "elapsed": 0.006778240203857422,
                            "initial": 0,
                            "n": 0,
                            "ncols": null,
                            "nrows": null,
                            "postfix": null,
                            "prefix": "TTTS-202303 openalex",
                            "rate": null,
                            "total": 21,
                            "unit": "it",
                            "unit_divisor": 1000,
                            "unit_scale": false
                        },
                        "application/vnd.jupyter.widget-view+json": {
                            "model_id": "54c2d6f117994ff1b3f3c2b6a8cfb82b",
                            "version_major": 2,
                            "version_minor": 0
                        },
                        "text/plain": [
                            "TTTS-202303 openalex:   0%|          | 0/21 [00:00<?, ?it/s]"
                        ]
                    },
                    "metadata": {},
                    "output_type": "display_data"
                }
            ],
            "source": [
                "pubmedCSVs = sorted(glob.glob(RESULTS_DIR))\n",
                "\n",
                "for pubmedCSV in pubmedCSVs:\n",
                "    topicdate = Path(pubmedCSV).stem\n",
                "\n",
                "    logger.info(\"Reading %s search results from PubMed CSV\", topicdate)\n",
                "    \n",
                "    pmresults_df = pd.read_csv(\n",
                "        pubmedCSV, \n",
                "        usecols=[0,1, 5, 6, 10], \n",
                "        names=[\"pmid\", \"title\", \"journal\", \"pubdate\", \"doi\"])\n",
                "\n",
                "    # add columns of helpful URLs\n",
                "    # hyperlinked URLs makes publications more accessible in the XLSX output\n",
                "\n",
                "    # DOI link, blank if no DOI\n",
                "    pmresults_df[\"doi\"] = [\n",
                "        f'=HYPERLINK(\"https://doi.org/{doi}\")' if pd.notna(doi) else \"\"\n",
                "        for doi in pmresults_df[\"doi\"]\n",
                "    ]\n",
                "\n",
                "    # Link to PubMed record\n",
                "    pmresults_df[\"pubmed\"] = [\n",
                "        f'=HYPERLINK(\"https://pubmed.ncbi.nlm.nih.gov/{pmid}\", {pmid})'\n",
                "        for pmid in pmresults_df[\"pmid\"]\n",
                "    ]\n",
                "\n",
                "    # Link to OHSU Library Catalog lookup for full-text access\n",
                "    pmresults_df[\"ohsu_library\"] = [\n",
                "        f'=HYPERLINK(\"https://librarysearch.ohsu.edu/openurl/OHSU/OHSU?sid=Entrez:PubMed&id=pmid:{pmid}\", \"Find @ OHSU\")'\n",
                "        for pmid in pmresults_df[\"pmid\"]\n",
                "    ]\n",
                "\n",
                "    # Link to Rush University Library Catalog lookup for full-text access\n",
                "    pmresults_df[\"rush_library\"] = [\n",
                "        f'=HYPERLINK(\"https://i-share-rsh.primo.exlibrisgroup.com/openurl/01CARLI_RSH/01CARLI_RSH:CARLI_RSH?sid=Entrez:PubMed&id=pmid:{pmid}\", \"Find @ Rush\")'\n",
                "        for pmid in pmresults_df[\"pmid\"]\n",
                "    ]\n",
                "\n",
                "    # \"Long PMID\" for MAGID mapping in OpenAlex\n",
                "    pmresults_df[\"long_pmid\"] = [\n",
                "        f\"https://pubmed.ncbi.nlm.nih.gov/{pmid}\" for pmid in pmresults_df[\"pmid\"]\n",
                "    ]\n",
                "\n",
                "    pmresults_df.set_index(\"long_pmid\", inplace=True)\n",
                "    pmresults_df.sort_index(inplace=True)\n",
                "    pmresults_df.to_csv(f\"{CSV_DIR}{topicdate}.csv\")\n",
                "    pmresults_df = pd.read_csv(f\"{CSV_DIR}{topicdate}.csv\", index_col=\"long_pmid\")\n",
                "\n",
                "    logger.info(\"Mapping PMIDs to MAGIDs\")\n",
                "    mags_df = pmid2magid(pmresults_df, topicdate)\n",
                "\n",
                "    logger.info(\"Joining PubMed search results with MAGID mapping\")\n",
                "    articles_df = mags_df.join(pmresults_df).set_index(\"magid\").sort_index()\n",
                "    articles_df.index = articles_df.index.astype(\"int64\")\n",
                "\n",
                "    logger.info(\"Scoring articles\")\n",
                "    articles_cited_df, articles_develop_df, articles_disrupt_df = score_results(articles_df, topicdate)\n",
                "\n",
                "    logger.info(\"Styling output XSLX\")\n",
                "    style_output(articles_cited_df, articles_develop_df, articles_disrupt_df, topicdate)"
            ]
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "fsdp-venv",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.9.13"
        },
        "orig_nbformat": 4
    },
    "nbformat": 4,
    "nbformat_minor": 2
}