vis-letsplot.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(vis-letsplot)=\n",
    "# Easy Data Visualisation for Tidy Data with **Lets-Plot**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "Here you'll see how to make plots quickly using the *declarative* plotting package **[lets-plot](https://lets-plot.org/)**. This package is perfect if you want to make a standard chart from so-called *tidy* data where you have one row per observation and one columnn per variable. This chapter has benefitted from the book *ggplot: elegant graphics for data analysis*.\n",
    "\n",
    "```{note}\n",
    "**lets-plot** is the quickest way to get going with plots in Python.\n",
    "```\n",
    "\n",
    "## Preliminaries\n",
    "\n",
    "To install **lets-plot**, run `pip install lets-plot` on the command line. We're also going to be using the Palmer Penguins dataset, so you'll need to run `pip install palmerpenguins` too.\n",
    "\n",
    "There is some background information that you might find useful in getting to grips with **lets-plot**. All plots are composed of the data, the information you want to visualise, and a mapping: the description of how the data’s variables are mapped to aesthetic attributes. There are five mapping components:\n",
    "\n",
    "- A *layer* is a collection of geometric elements and statistical transformations. Geometric elements, *geoms* for short, represent what you actually see in the plot: points, lines, polygons, etc. Statistical transformations, *stats* for short, summarise the data: for example, binning and counting observations to create a histogram, or fitting a linear model.\n",
    "\n",
    "- *Scales* map values in the data space to values in the aesthetic space. This includes the use of colour, shape or size. Scales also draw the legend and axes.\n",
    "\n",
    "- A *coord*, or coordinate system, describes how data coordinates are mapped to the plane of the graphic. It also provides axes and gridlines to help read the graph. We normally use the Cartesian coordinate system, but a number of others are available, including polar coordinates and map projections.\n",
    "\n",
    "- A *facet* specifies how to break up and display subsets of data as small multiples.\n",
    "\n",
    "- A *theme* controls the finer points of display, like the font size and background colour. While the defaults have been chosen with care, you may need to consult other references to create an attractive plot.\n",
    "\n",
    "As ever, we're going to load the packages we'll be using."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from lets_plot import *\n",
    "from palmerpenguins import load_penguins\n",
    "\n",
    "LetsPlot.setup_html()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll notice the two main quirks of **lets-plot** already: the first is that we imported *everything* in the package using `import *`. This is to make it easier to use the package on-the-fly, because it has a lot of named functions. Second, we ran `LetsPlot.setup_html()`; this allows **lets-plot** charts to be displayed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting started with **lets-plot**\n",
    "\n",
    "The goal of this section is to teach you how to produce useful graphics with **lets-plot** as quickly as possible. We're going to cover:\n",
    "\n",
    "-   The three key components of **lets-plot** chart: data, aesthetics and geoms\n",
    "\n",
    "-   How to add additional variables to a plot with aesthetics\n",
    "\n",
    "-   How to display additional categorical variables in a plot using small multiples created by faceting\n",
    "\n",
    "-   A variety of different *geoms* that you can use to create different types of plots\n",
    "\n",
    "-   How to modify the axes\n",
    "\n",
    "-   Things you can do with a plot object other than display it in your interactive window, like save it to disk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load our data into **pandas**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "penguins = load_penguins()\n",
    "penguins.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basics\n",
    "\n",
    "Every plot has three key components: data, aesthetic mappings, layers (at least one, called geoms). Here's a simple example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(penguins, aes(x=\"body_mass_g\", y=\"flipper_length_mm\")) + geom_point())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This produces a scatterplot defined by those three elements: the *data* are from the `penguin` dataframe, the *aesthetic mapping* is x to body size in grams and flipper length in mm to the y position, and finally the layer or *geom* with points.\n",
    "\n",
    "Note that data and aesthetic mappings were supplied to a function called `ggplot`, which accepts the *data* and *aes*, then layers/*geoms* are added on with +. The pattern will be similar for all **lets-plot** charts. In this case, we modified the *geom* to have slightly larger points by setting `size=5`, but we could have just left it with the default setting. \n",
    "\n",
    "Note that the variables `x` and `y` in the `aes` call are necessary positional arguments, so you can simply omit saying `x=` and `y=` like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(penguins, aes(\"body_mass_g\", \"flipper_length_mm\")) + geom_point())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding extra dimensions: shape, colour, and size\n",
    "\n",
    "Although you should always be careful not to put too much information on a chart, you can add further dimensions to these plots. Let's demonstrate this by adding colour to the mix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(penguins, aes(\"body_mass_g\", \"flipper_length_mm\", colour=\"island\"))\n",
    "    + geom_point()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that this has rendered the categorical variable \"island\" by having it appear in different colours. A legend has automatically been added. Do remember that not everyone can see all colours well, so it's best to use colourblind-friendly colour scales whenever possible.\n",
    "\n",
    "```{note}\n",
    "Be careful with data types when adding extra dimensions to charts: if your data type is float or int instead of categorical, you will get a continous colour gradient instead of a discrete colour scale.\n",
    "```\n",
    "\n",
    "Let's look at shape too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(penguins, aes(\"body_mass_g\", \"flipper_length_mm\", shape=\"island\"))\n",
    "    + geom_point()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although we previously set the size of the points overall, we can use them as an aesthetic too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(penguins, aes(\"body_mass_g\", \"flipper_length_mm\", size=\"island\"))\n",
    "    + geom_point(alpha=0.5)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above, we used `alpha=0.5`, which is a transparency setting, to make it easier to see overlapping points.\n",
    "\n",
    "And just as we previously set the size of the points to be a single, universal value, we can do the same for shape and colour—we just need to set it in `geom_point()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Facets\n",
    "\n",
    "You can use facets (aka small multiples) to display more dimensions of information too. To facet your plot by a single variable, use `facet_wrap()`. The first argument of `facet_wrap()` tells the function what variable to have in successive charts. The variable that you pass to `facet_wrap()` should be categorical."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(penguins, aes(\"body_mass_g\", \"flipper_length_mm\"))\n",
    "    + geom_point()\n",
    "    + facet_wrap(facets=\"island\", ncol=3)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot Geoms\n",
    "\n",
    "By substituting `geom_point()` for a different geom function, you'll get a different type of plot. You're now going to see some of the other important *geoms* provided in **lets-plot**.\n",
    "\n",
    "-   `geom_smooth()` fits a smoothed conditional line then plots it and its standard error.\n",
    "\n",
    "-   `geom_boxplot()` produces a box-and-whisker plot to summarise the distribution of a set of points.\n",
    "\n",
    "-   `geom_histogram()` and `geom_density()` show the distribution of continuous variables.\n",
    "\n",
    "-   `geom_bar()` shows counts of categorical variables.\n",
    "\n",
    "-   `geom_path()` and `geom_line()` draw lines between the data points.\n",
    "    A line plot is constrained to produce lines that travel from left to right, while paths can go in any direction.\n",
    "    Lines are typically used to explore how things change over time.\n",
    "\n",
    "Let's take a closer look at some of these:\n",
    "\n",
    "### Fitting a line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(penguins, aes(\"body_mass_g\", \"flipper_length_mm\"))\n",
    "    + geom_point()\n",
    "    + geom_smooth(method=\"loess\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use a linear model instead with `method=\"lm\"` (this is the default)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Jittered points and boxplots\n",
    "\n",
    "These are especially useful when we have lots of data that overlap, or want to get more of an idea of the overall distribution, or both."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(penguins, aes(\"island\", \"body_mass_g\")) + geom_jitter())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Box plots are created via:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(penguins, aes(\"island\", \"body_mass_g\")) + geom_boxplot())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Histograms and probability density plots\n",
    "\n",
    "You're probably getting a good idea of how this works now! Here are the geoms for histograms and probability density plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(penguins, aes(\"body_mass_g\")) + geom_histogram())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`geom_histogram()` has a `bins=` keyword argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(penguins, aes(\"body_mass_g\")) + geom_density())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember, as ever, you can use `help(FUNCTIONNAME)` to get help on the options and keyword arguments for any function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bar Charts\n",
    "\n",
    "These are as you'd expect, but if you don't want a count of the number of items but just to display the given values, you can use the keyword argument `stat=\"identity\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(penguins, aes(\"species\")) + geom_bar())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Line charts and time series\n",
    "\n",
    "Let's grab some data with a time dimension from FRED: vacancies and unemployment percent in the USA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "\n",
    "import pandas_datareader.data as web\n",
    "\n",
    "start = datetime.datetime(2000, 1, 1)\n",
    "end = datetime.datetime(2021, 1, 1)\n",
    "code_dict = {\n",
    "    \"Vacancies\": \"LMJVTTUVUSA647N\",\n",
    "    \"Unemployment\": \"UNRATE\",\n",
    "    \"LabourForce\": \"CLF16OV\",\n",
    "}\n",
    "list_dfs = [\n",
    "    web.DataReader(value, \"fred\", start, end)\n",
    "    .rename(columns={value: key})\n",
    "    .groupby(pd.Grouper(freq=\"YS\"))\n",
    "    .mean()\n",
    "    for key, value in code_dict.items()\n",
    "]\n",
    "vu_data = pd.concat(list_dfs, axis=1)\n",
    "vu_data = vu_data.assign(\n",
    "    Vacancies=100 * vu_data[\"Vacancies\"] / (vu_data[\"LabourForce\"] * 1e3)\n",
    ").dropna()\n",
    "vu_data[\"Date\"] = vu_data.index\n",
    "vu_data[\"Year\"] = vu_data.index.year\n",
    "vu_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(vu_data, aes(\"Date\", \"Vacancies\")) + geom_line(size=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can make this even more interesting by looking at how two variables have co-moved in time together with a connected scatter plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(vu_data, aes(\"Unemployment\", \"Vacancies\"))\n",
    "    + geom_path(size=1)\n",
    "    + geom_point(size=5)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Labels and Titles\n",
    "\n",
    "`xlab()` and `ylab()` modify the x- and y-axis labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(penguins, aes(\"body_mass_g\", \"flipper_length_mm\"))\n",
    "    + geom_point()\n",
    "    + xlab(\"Body mass (g)\")\n",
    "    + ylab(\"Flipper length (mm)\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But you can also specify all labels and titles at once like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(penguins, aes(x=\"flipper_length_mm\", y=\"body_mass_g\"))\n",
    "    + geom_point(aes(color=\"species\", shape=\"species\"))\n",
    "    + geom_smooth(method=\"lm\")\n",
    "    + labs(\n",
    "        title=\"Body mass and flipper length\",\n",
    "        subtitle=\"Dimensions for Adelie, Chinstrap, and Gentoo Penguins\",\n",
    "        x=\"Flipper length (mm)\",\n",
    "        y=\"Body mass (g)\",\n",
    "        color=\"Species\",\n",
    "        shape=\"Species\",\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding text annotations\n",
    "\n",
    "Should you wish to add text annotations to your plots, you can!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(vu_data, aes(\"Unemployment\", \"Vacancies\"))\n",
    "    + geom_path(size=1, color=\"gray\")\n",
    "    + geom_point(color=\"gray\", size=5)\n",
    "    + geom_text(aes(label=\"Year\"), position=position_nudge(y=0.3))\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Limits on axes\n",
    "\n",
    "You always have an option when it comes to removing points from your data: you can filter your dataframe or change the limits on your axes when you are plotting data. If you wish to do the latter, use the `xlim` and `ylim` comnands to do this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(penguins, aes(x=\"flipper_length_mm\", y=\"body_mass_g\"))\n",
    "    + geom_point(size=4)\n",
    "    + xlim(200, 230)\n",
    "    + ylim(3e3, 5e3)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other useful-to-know elements of **lets-plot** charts\n",
    "\n",
    "We don't want to go into every detail of **lets-plot** here, as the [documentation](https://lets-plot.org/) is absolutely excellent and comprehensive—so you can find whatever you need there. But it may be useful to at least know of some further features we didn't look at here, such as:\n",
    "\n",
    "- changing the theme and look of a plot\n",
    "- changing the scales (eg the axis ticks)\n",
    "- maps and geospatial charts\n",
    "- sampling\n",
    "- contour and other plots that show three dimensions via a level set, $ z = f(x, y) $."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving your plots to file\n",
    "\n",
    "Once you've made a plot, you might want to save it as an image that you can use elsewhere.\n",
    "That's the job of `ggsave()`, which will save the plot most recently created to disk:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plotted_data = (\n",
    "    ggplot(penguins, aes(x=\"flipper_length_mm\", y=\"body_mass_g\")) + geom_point()\n",
    ")\n",
    "ggsave(plotted_data, filename=\"penguin-plot.svg\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This saved the figure to disk at the location shown—by default it's in a subdirectory called \"lets-plot-images\".\n",
    "\n",
    "We used the file format \"svg\". There are lots of output options to choose from to save your file to. Remember that, for graphics, *vector formats* are generally better than *raster formats*. In practice, this means saving plots in svg or pdf formats over jpg or png file formats. The svg format works in a lot of contexts (including Microsoft Word) and is a good default. To choose between formats, just supply the file extension and the file type will change automatically, eg \"chart.svg\" for svg or \"chart.png\" for png. You can also save figures in HTML format.\n",
    "\n",
    "If you're using a raster format then you'll need to specify how big the figure is via the *scale* keyword argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "import shutil\n",
    "\n",
    "shutil.rmtree(\"lets-plot-images\")"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "codeforecon",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}