BAVAPI Jupyter Demo.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `bavapi` Jupyter notebook demo\n",
    "\n",
    "In this notebook we will demo the functionality of `bavapi`, a python package to access BAV data from the WPPBAV [Fount](https:/fount.wppbav.com).\n",
    "\n",
    "To learn more about `bavapi` and how to use it, please visit the [documentation](https://wppbav.github.io/bavapi-sdk-python/).\n",
    "\n",
    "To learn more about the `BAV API` and how to use it, please visit the [API documentation](https://developer.wppbav.com)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Motivation\n",
    "\n",
    "The reason for using `bavapi` is that, while the Fount API provides a way to programmatically access BAV data, it is primarily intended for integrating BAV data into web applications.\n",
    "\n",
    "This means that the way the data is provided makes it a bit harder to use in data science analysis:\n",
    "\n",
    "- The data is delivered in JSON format, which has to be parsed into a DataFrame.\n",
    "- Some of the data is composed of nested dictionaries, which makes it a bit harder to parse.\n",
    "- The data is delivered in a paginated form, meaning that it is chunked into batches of 25 rows by default. A typical BSS would take about 35 requests to get all data at 25 rows per request.\n",
    "\n",
    "`bavapi` provides a simple solution to get this data in a workable format for data science workflows."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fount API Token\n",
    "\n",
    "A Fount API bearer token (aka key) is required to use `bavapi`, which you can retrieve from your [Fount](https:/fount.wppbav.com) account settings under the \"API\" tab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TOKEN = \"your_token\"  # paste your token here, or follow the instructions below"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cells uses `python-dotenv` ([learn more](https://github.com/theskumar/python-dotenv)) to load the token from a `.env` file, as recommended in the `bavapi` [documentation](https://wppbav.github.io/bavapi-sdk-python/latest/getting-started/authentication/)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">__**💡 Tip:**__ Uncomment the cell below to install `python-dotenv`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %pip install python-dotenv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "load_dotenv()\n",
    "TOKEN = os.environ[\"FOUNT_API_KEY\"]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using `bavapi` to get BAV data\n",
    "\n",
    "To get started, install `bavapi` by running the following command:\n",
    "\n",
    "```\n",
    "pip install wpp-bavapi\n",
    "```\n",
    "\n",
    "For more information, see the [installation](https://wppbav.github.io/bavapi-sdk-python/latest/getting-started/installation/) instructions in the `bavapi` documentation."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">__**💡 Tip:**__ Uncomment the cell below to install `bavapi`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %pip install wpp-bavapi"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have installed `bavapi`, import it and start using one of the available endpoint functions:\n",
    "\n",
    "- `audiences`\n",
    "- `brand_metric_groups`\n",
    "- `brand_metrics`\n",
    "- `brands`\n",
    "- `brandscape_data`\n",
    "- `categories`\n",
    "- `collections`\n",
    "- `sectors`\n",
    "- `studies`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import bavapi"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Brands endpoint\n",
    "\n",
    "Now we can query the database. For example, we can request which brands exist with `\"Swatch\"` in their name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "swatch = bavapi.brands(TOKEN, \"Swatch\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And get a DataFrame with all available information for brands that contain \"Swatch\" in their name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "swatch"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Studies endpoint\n",
    "\n",
    "Similarly, you can query the `studies` endpoint to get a summary of all BAV studies, with advanced filtering capabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "uk_studies = bavapi.studies(TOKEN, \"GB\", 2022)\n",
    "uk_studies"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Audiences endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "audiences = bavapi.audiences(TOKEN, active=1)\n",
    "audiences.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Brandscape data endpoint\n",
    "#### Valid queries\n",
    "\n",
    "The `brandscape-data` endpoint has been restricted to require at least one of these specific set of filters:\n",
    "\n",
    "- Study + Audience + Brand + Category\n",
    "- Country + Year + Audience\n",
    "- Brand + Audience + Country + Year\n",
    "\n",
    "You should read these from left to right. A combination of \"Study + Audience\" works just as well as \"Study + Audience + Brand\". However, \"Category + Audience\" will not.\n",
    "\n",
    "If you use Country or Year filters, you must use both filters together.\n",
    "\n",
    "If a query does not have any of these combinations of filters, it will raise a `ValidationError`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "uk_bss = bavapi.brandscape_data(TOKEN, audiences=1)  # raises ValidationError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "uk_bss = bavapi.brandscape_data(TOKEN, country_code=\"GB\", year_number=2020, audiences=1)\n",
    "uk_bss.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Downloading and using Reference Classes\n",
    "\n",
    "The `brandscape_data` endpoint will always return all audiences (aka bases) for a study unless audiences are specified. This means queries without base information can require substantially more requests (~35 audiences, a typical study with all audiences would need over 500 total requests).\n",
    "\n",
    "To more easily use this endpoint, you can download the reference classes for Audiences and Countries, which will make filtering more explicit.\n",
    "\n",
    "For example, normally you would have to specify the audience by using its Fount ID. This can be retrieved from the `audiences` endpoint, and used like so:\n",
    "\n",
    "```py\n",
    "bavapi.brandscape_data(TOKEN, country_code=\"GB\", audiences=123)\n",
    "```\n",
    "\n",
    "Or alternatively, use the reference class:\n",
    "\n",
    "```py\n",
    "bavapi.brandscape_data(TOKEN, country_code=\"GB\", audiences=Audiences.ALL_ADULTS)\n",
    "```\n",
    "\n",
    "To download these references, you must specify your Fount API key in a `.env` file at the top of your current working directory.\n",
    "\n",
    "Run the following to generate reference files in a new directory `bavapi_refs` (replacing `\"TOKEN\"` with your token):\n",
    "\n",
    "```prompt\n",
    "bavapi-gen-refs --all -t \"TOKEN\"\n",
    "```\n",
    "\n",
    ">__**⚠ WARNING:**__ DO NOT PUSH REFERENCE FILES TO GIT! Add `bavapi_refs/` to your `.gitignore` file."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">__**💡 Tip:**__ Uncomment the cell below to download reference files. Replace `TOKEN` with your API key. If you have set up a `.env` file with `FOUNT_API_KEY` as an environment variable, you can use the following cell instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !bavapi-gen-refs --all -t \"TOKEN\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !bavapi-gen-refs --all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you should be ready to import reference classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bavapi_refs import Audiences"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same can be done with countries. Use:\n",
    "\n",
    "```py\n",
    "from bavapi_refs import Countries\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More examples\n",
    "\n",
    "Here are some additional examples of more complex queries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get one single brand by ID\n",
    "swatch = bavapi.brands(\n",
    "    TOKEN, brand_id=2985  # Note that this ID is the Fount ID, not the one in `bav_key`\n",
    ")\n",
    "swatch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get US studies that ended fielding after `2021-09-01`\n",
    "us_studies = bavapi.studies(\n",
    "    TOKEN, \"US\", filters={\"data_updated_since\": \"2021-09-01 00:00:00\"}\n",
    ")\n",
    "us_studies.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get Spanish data for 2022 for the `Adults with kids` base sorted by Differentiation in descending order (highest first)\n",
    "es_bss = bavapi.brandscape_data(\n",
    "    TOKEN,\n",
    "    \"ES\",\n",
    "    2022,\n",
    "    [Audiences.ADULTS_WITH_KIDS, Audiences.ADULTS_WITH_KIDS_0_11],\n",
    "    sort=\"-differentiation_rank\",\n",
    ")\n",
    "es_bss.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get Spanish data for 2020 and 2022\n",
    "es_bss = bavapi.brandscape_data(TOKEN, \"ES\", [2019, 2022], Audiences.ALL_ADULTS)\n",
    "es_bss.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get French and Spanish data for 2022\n",
    "es_fr_bss = bavapi.brandscape_data(TOKEN, [\"ES\", \"FR\"], 2022, Audiences.ALL_ADULTS)\n",
    "es_fr_bss.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using `bavapi.Query` with endpoint functions\n",
    "\n",
    ">__New in `v0.11.0`__.\n",
    "\n",
    "Instead of passing parameters to the functions, you can also create a `bavapi.Query` object and use it instead.\n",
    "\n",
    "One benefit of doing this is that you could reuse the `Query`, or even save as a file and load it for reproducibility."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a `Query` object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = bavapi.Query(\n",
    "    filters=bavapi.filters.StudiesFilters(\n",
    "        country_codes=[\"US\", \"GB\"], year_numbers=2023\n",
    "    ),\n",
    "    include=\"country\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use it in the `studies` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "anglo_studies = bavapi.studies(TOKEN, query=query)\n",
    "anglo_studies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can save it as a JSON file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"query.json\", \"w\", encoding=\"utf-8\") as file:\n",
    "    file.write(query.model_dump_json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And load again for later reuse:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "with open(\"query.json\", encoding=\"utf-8\") as file:\n",
    "    query = bavapi.Query(**json.load(file))\n",
    "\n",
    "loaded_anglo_studies = bavapi.studies(TOKEN, query=query)\n",
    "loaded_anglo_studies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">__**Note:**__ The loaded filter object will be of type `FountFilters` instead of the original type, but the requests should work as intended."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Raw Queries\n",
    "\n",
    "If we wanted to perform a request where we get the raw json data for some reason, that is also possible within `bavapi`.\n",
    "\n",
    "However, we will need to use the `Query` object in conjunction with the `raw_query` function to make the query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_res = bavapi.raw_query(\n",
    "    TOKEN,\n",
    "    \"companies\",\n",
    "    bavapi.Query(filters=bavapi.filters.FountFilters(name=\"Meta\")),\n",
    ")\n",
    "raw_res[0]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced Usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Control pagination and batching of requests\n",
    "\n",
    "`bavapi` offers the option to control the behavior of the batching and error handling processes. These are determined by the following endpoint parameters:\n",
    "\n",
    "- `page`: The page number to request (if `per_page` and `max_pages` are not set) or the page number to start the pagination from, default `None`\n",
    "- `per_page`: The number of elements in each page, default `100`\n",
    "- `max_pages`: The maximum number of pages to request (if the total number of pages exceeds `max_pages`), default `None`\n",
    "- `batch_size`: The number of pages to include in each batch, default `10`\n",
    "- `n_workers`: The number of workers to use for the batching process, default `2`\n",
    "- `retries`: The number of times to retry a request if it fails, default `3`\n",
    "- `on_errors`: Whether to warn about failed pages (default) or raise an exception immediately if the request fails, default `\"warn\"`\n",
    "\n",
    ">_**NOTE:**_ The default values for these parameters are set so that there are about 20 concurrent requests to the API. More than 30 concurrent requests may result in more failures. Use with caution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "res = bavapi.brands(\n",
    "    TOKEN,\n",
    "    studies=546,\n",
    "    page=2,  # Start with page 2\n",
    "    per_page=25,  # Retrieve 25 elements per page\n",
    "    max_pages=20,  # Retrieve 20 pages or all pages, whichever is smaller\n",
    "    batch_size=5,  # Request 5 pages in each batch\n",
    "    n_workers=4,  # Use 4 workers\n",
    "    retries=2,  # Retry twice on failures\n",
    "    on_errors=\"raise\",  # Raise on errors immediately, instead of issuing a warning\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Asynchronous requests\n",
    "\n",
    ">_**NOTE:**_ This section is more geared towards using `bavapi` in other libraries or applications, or when having to make many separate queries. For general Jupyter notebook use, the examples above should work in most cases.\n",
    "\n",
    "`bavapi` provides a `Client` class that provides a direct `async` interface for developing asynchronous programs or when having to make many separate queries.\n",
    "\n",
    "`bavapi.Client` is an asynchronous context manager, similar in use to `httpx.AsyncClient` (which powers `bavapi`) or `requests.Session`.\n",
    "\n",
    "It also brings performance benefits and more. See the [documentation](https://wppbav.github.io/bavapi-sdk-python/latest/usage/advanced/) for more details.\n",
    "\n",
    "The `Client` can be used directly in Jupyter notebooks, or within asynchronous functions (defined with `async def`).\n",
    "\n",
    "Use the `async with` context manager syntax, and `await` the desired endpoint method like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "async with bavapi.Client(TOKEN) as client:\n",
    "    fr_bss = await client.brandscape_data(\n",
    "        country_code=\"FR\",\n",
    "        year_number=2022,\n",
    "        audiences=Audiences.ALL_ADULTS,\n",
    "    )\n",
    "fr_bss.head()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "ead85fe90e8fb85766f3ec805e9c8de16267b7b912249df01b46833aac76b7ab"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}