Research_Outputs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pure API Demonstration: Research Outputs\n",
    "\n",
    "These notebooks demonstrate some uses of the API of Elsevier's *Pure* Current Research Information System (CRIS). This notebook demonstrates some requests for research outputs.\n",
    "\n",
    "**Enter API details - including an API key which gives access to the `research-outputs` endpoint - in [`_Config_DO_THIS_FIRST.ipynb`](./_Config_DO_THIS_FIRST.ipynb) and execute that notebook before executing this notebook.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We're using the requests library to talk to the API\n",
    "import requests\n",
    "\n",
    "# The display, HTML and Markdown libraries will help render HTML and Markdown\n",
    "from IPython.core.display import display, HTML, Markdown\n",
    "\n",
    "# The utility_functions.py script includes:\n",
    "# - pretty_print_json(json_object, ind=4) - prints json with indentation and colours\n",
    "import utility_functions as uf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieve the api_url and headers set in the config notebook\n",
    "%store -r api_url\n",
    "%store -r headers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We'll be making requests to /research-outputs\n",
    "request_url = \"/\".join([api_url,\"research-outputs\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple unqualified request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make request with no parameters\n",
    "# By default, returns most recent research outputs in descending date order\n",
    "response = requests.get(url=request_url, headers=headers)\n",
    "research_outputs_json = response.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display raw output\n",
    "uf.pretty_print_json(research_outputs_json)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print the UUIDs and titles of all returned items\n",
    "for item  in research_outputs_json[\"items\"]:\n",
    "    print(item[\"uuid\"], \":\", item[\"title\"][\"value\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Searching\n",
    "\n",
    "We can search with the `q` parameter in a `GET` request, using the [Lucene syntax](\"https://lucene.apache.org/core/2_9_4/queryparsersyntax.html\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make request to /research-outputs passing query terms via the 'q' parameter\n",
    "# Let's look for research outputs which mention \"software\" or \"programming\" in their title, subtitle or abstract\n",
    "request_params = {\"q\" : [\"(^title:(software OR programming) OR ^subTitle:(software OR programming) OR ^abstract:(software OR programming))\"]}\n",
    "response = requests.get(url=request_url, headers=headers, params=request_params)\n",
    "research_outputs_json = response.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display raw output\n",
    "uf.pretty_print_json(research_outputs_json)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get only specified fields"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the request parameters specifying the desired fields:\n",
    "# - pureId\n",
    "# - uuid\n",
    "# - title\n",
    "# - open access status\n",
    "# - names of associated persons\n",
    "request_params = {\"fields\" : [\"pureId\", \"uuid\", \"title\", \"openAccessPermission.value\", \"personAssociations.name.*\"]}\n",
    "response = requests.get(url=request_url, headers=headers, params=request_params)\n",
    "research_outputs_json = response.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display raw output\n",
    "uf.pretty_print_json(research_outputs_json)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get HTML renderings\n",
    "\n",
    "Note that when requesting specific renderings, almost all other fields will not be included in the response unless explicitly requested using the \"fields\" parameter as above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the request parameters - specifying standard, harvard and apa renderings\n",
    "request_params = {\"rendering\" : [\"standard\", \"harvard\", \"apa\"]}\n",
    "response = requests.get(url=request_url, headers=headers, params=request_params)\n",
    "research_outputs_json = response.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display raw output\n",
    "uf.pretty_print_json(research_outputs_json)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Render all HTML\n",
    "for item in research_outputs_json[\"items\"]:\n",
    "    # Display the format and render the content of each rendering\n",
    "    for r in item[\"renderings\"]:\n",
    "        display(HTML(\"<h4>\" + r[\"format\"] + \"</h4>\"))\n",
    "        display(HTML(r[\"html\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## POST requests\n",
    "\n",
    "A greater range of parameters are available to be passed in the body of POST requests (using JSON or XML format) than can be passed as GET parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We need the json library to create the POST body\n",
    "import json\n",
    "\n",
    "# We're going to get a count of research outputs published each year from 2001 to 2018\n",
    "pub_counts = {}\n",
    "\n",
    "for year in range(2001, 2019):\n",
    "    # Create the POST body as a dictionary and serialize as JSON (json.dumps)\n",
    "    # \"size\" = 0 because we're only interested in the count at the start of the response\n",
    "    # - don't need to see any individual records\n",
    "    post_body = json.dumps({\"size\": 0, \"publishedAfterDate\": f'{year}-01-01', \"publishedBeforeDate\": f'{year + 1}-01-01'})\n",
    "    # We need to modify the headers to specify the type of data we're submitting\n",
    "    post_headers = headers.copy()\n",
    "    post_headers[\"Content-Type\"] = 'application/json'\n",
    "    # Make the request\n",
    "    response = requests.post(url=request_url, headers=post_headers, data=post_body)\n",
    "    research_outputs_json = response.json()\n",
    "    # Add the result for this year to the results dictionary\n",
    "    pub_counts[year] = research_outputs_json[\"count\"]\n",
    "\n",
    "print(pub_counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualising this data\n",
    "\n",
    "We're going to use the Bokeh library to visualise this data. We won't go into the details of using Bokeh here; it's presented as an example of what can be done with data from the API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bokeh.io import show, output_notebook\n",
    "from bokeh.plotting import figure\n",
    "from bokeh.palettes import PuBu\n",
    "\n",
    "output_notebook()\n",
    "\n",
    "# x labels need to be strings\n",
    "x_labels = list(map(str, list(pub_counts.keys())))\n",
    "y_values = list(pub_counts.values())\n",
    "\n",
    "p = figure(x_range=x_labels, plot_height=500, title=\"Publication Counts by Year\")\n",
    "p.vbar(x=x_labels, top=y_values, width=0.9, color=PuBu[7][2])\n",
    "\n",
    "p.xgrid.grid_line_color = None\n",
    "p.y_range.start = 0\n",
    "\n",
    "show(p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}