04-Python-Bliss.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c1b38714",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Advanced HDF5 with h5py Tutorial\n",
    "\n",
    "#### Prepared for 2023 HDF5 User Group Meeting\n",
    "\n",
    "\n",
    "Aleksandar Jelenak (<ajelenak@hdfgroup.org>)</br>\n",
    "https://orcid.org/0009-0001-2102-0559"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b5a82b1",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# About h5py\n",
    "\n",
    "* Created by Andrew Collette in 2008\n",
    "* Andrew also wrote a book about h5py: https://www.oreilly.com/library/view/python-and-hdf5/9781491944981\n",
    "* The most popular package for using HDF5 library (libhdf5) in Python\n",
    "* As of 2023-08-13, the h5py GitHub repository had 1,912 stars"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03a4dded",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# What makes h5py popular?\n",
    "\n",
    "* **NumPy array** ↔︎ h5py ↔︎ bytes ↔︎ libhdf5 ↔︎ **HDF5 file**\n",
    "* API natural to Python users\n",
    "* Very smart transition to the libhdf5 API"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8125cf11",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# h5py _High-Level_ (Pythonic) API\n",
    "\n",
    "* h5py classes for HDF5 entities: Group, Dataset, Commited Datatype, Attribute.\n",
    "* h5py File also represents the root group (so is an h5py Group, too).\n",
    "* h5py Group and Attribute classes based on the Python dictionary.\n",
    "* HDF5 links (names of HDF5 objects in a group) are the keys of a group or attribute object dictionaries."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0570495",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "* Also high-level classes for Virtual Datasets, filters, hyperslab selections, and dimension scales."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18eabf75",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# h5py _Low-Level_ API\n",
    "\n",
    "* Gateway to the libhdf5 C API\n",
    "* Easy to find the h5py equivalent of any libhdf5 C function. Example: `H5Dget_storage_size` ➞ `h5py.h5d.get_storage_size`.\n",
    "* Applies to libhdf5 constants, too: `H5F_LIBVER_EARLIEST` ➞ `h5py.h5f.LIBVER_EARLIEST`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eed1ae87",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## How to Access Low-Level API?\n",
    "\n",
    "* `id` property of the high-level h5py objects (File, Group, Dataset, Datatype).\n",
    "* For everything else (datatypes, property lists, etc.), there are special h5py classes. Example: `h5py.h5p.PropFCID` class for file creation property lists.\n",
    "* This API is very useful but is not meant to be used frequently. (Don't do C-style programming in Python.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b9dea86",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# h5py Resources\n",
    "\n",
    "* https://github.com/h5py/h5py\n",
    "* https://docs.h5py.org\n",
    "* https://api.h5py.org\n",
    "* Since 2023 is the Year of Open Science... Properly cite h5py to make your scientific work more [FAIR](https://www.go-fair.org/fair-principles/): https://doi.org/10.5281/zenodo.594310"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ece4159",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# h5py Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbf29603",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We'll start from the file `ou_process.h5` which is in the parent folder and make a copy in the current folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aec8f68a",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from shutil import copy\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe091af9",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "src_file = Path('./ou_process.h5')\n",
    "tgt_file = Path('./h5py_tutorial_ou_process.h5')\n",
    "copy(src_file, tgt_file)\n",
    "tgt_file.is_file()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae004016",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import h5py\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e47823c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# h5py File and Group Objects\n",
    "\n",
    "Open the copied file in the append mode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e764998",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "f = h5py.File(str(tgt_file), mode='a')\n",
    "f"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b0c72e9",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "isinstance(f, h5py.Group)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d88855b",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "f.filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64c70a71",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "f.name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2acfd476",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Low-level API for the File object\n",
    "\n",
    "These are the libhdf5 functions that take as the first argument a file identifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0318c83",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "f.id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9dd1b573",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "f.id.id"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "884a683a",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "libhdf5 `H5Fget_mdc_config` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a0104e3",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "h5ac_cache = f.id.get_mdc_config()\n",
    "h5ac_cache"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec6c5ab3",
   "metadata": {},
   "source": [
    "This object holds the file's metadata cache configuration."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de993348",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Important libhdf5 structs appear as Python extension types that provide property-style access to their struct fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4614a80b",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "h5ac_cache.version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b4599f9",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "h5ac_cache.dirty_bytes_threshold"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1ae4ddd",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The root group's (or any other HDF5 group) members are available through standard Python dictionary API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1eb43b81",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "list(f.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21a3dd43",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Access to HDF5 datasets is from the root group or their parent group, using the standard Python dictionary \"get key\" operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20ee515a",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "dset = f['dataset']\n",
    "dset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d650b261",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "HDF5 attributes are available from the `.attrs` property, again, with Python dictionary interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1131dfb4",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "list(dset.attrs.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ae544f4",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "This dataset does not have any attributes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80d7a69e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# h5py Dataset Object\n",
    "\n",
    "HDF5 datasets hold the file's data so there are many properties and methods to work with them. Reading/writing data supports NumPy slicing syntax and NumPy arrays:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc54e989",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "dset[10:12, 500:510]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ed5e1d4",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Most popular dataset settings are available as read-only properties:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50ddecbf",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "dset.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b2eb2d7",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(dset.chunks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cc0f7ef",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "dset.size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f4bf3b9",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "dset.nbytes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "252b11eb",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Low level API for the Dataset Object\n",
    "\n",
    "These are the libhdf5 functions that take as the first argument a dataset identifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5af394a6",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "dset.id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28971b04",
   "metadata": {},
   "outputs": [],
   "source": [
    "dset.id.id"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72fafd15",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let's check if the `dset` is chunked using the low-level API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aee09d71",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    dset  # h5py.Dataset\n",
    "        .id  # libhdf5 H5D API\n",
    "        .get_create_plist()  # H5Dget_create_plist -> h5py.h5p.PropDCID\n",
    "        .get_layout()  # H5Pget_layout\n",
    "    ==\n",
    "    h5py.h5d.CHUNKED  # H5D_CHUNKED\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7edf2c07",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We will now add new content to this file for use in the rest of the tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fe29027",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Working with Attributes\n",
    "\n",
    "We will add several attributes to the root group and the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3aebbbeb",
   "metadata": {},
   "outputs": [],
   "source": [
    "f.attrs['Conventions'] = 'HUG23'\n",
    "f.attrs['title'] = 'Content prepared for the HUG23 h5py tutorial'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b3f7e1b",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "dset.attrs['description'] = 'Sample dataset with some data'\n",
    "dset.attrs['version'] = 1\n",
    "dset.attrs['units'] = 'some_units'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "140668fe",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We will flush this new content to the file, just to be sure it is stored, and check these attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4484703",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "f.flush()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12612169",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "for n, v in f.attrs.items():\n",
    "    print(f'{n} = {v}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1190169b",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## h5py and HDF5 Strings\n",
    "\n",
    "The past is fraught with issues and several different ways of interpreting HDF5 strings. The current approach started with h5py v3.0.\n",
    "* HDF5 dataset strings:\n",
    "    * `bytes` objects if variable-length\n",
    "    * NumPy bytes arrays (`S` dtype) if fixed-length\n",
    "    * `h5py.Dataset.asstr()` to retrieve them as `str` objects\n",
    "\n",
    "* HDF5 attribute strings:\n",
    "    * `str` objects if variable-length\n",
    "    * NumPy bytes arrays if fixed-length"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "201d6b6e",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Note the difference in the `mu` and the `Conventions` attribute values above. We will now close the file in order to use `h5dump` tool to show the actual storage settings of these two attributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4fda7f2",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ff5bf87",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "!h5dump -a /mu -a /Conventions {str(tgt_file)}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e8f268b",
   "metadata": {},
   "source": [
    "What is different?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8cbf873",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We're going to reopen the file to continue adding more data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8fc17b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "f = h5py.File(str(tgt_file), mode='a')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ed639d5",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Dataset Filters\n",
    "\n",
    "The data of chunked datasets can be modified on-the-fly by _filters_. The most popular filters are for data compression. Registered filters are listed at https://portal.hdfgroup.org/display/support/Registered+Filter+Plugins.\n",
    "\n",
    "A dataset chunk is a fixed-size part of the dataset with the same rank as the dataset. Chunks do not overlap and are also known as _tiles_."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "070ff3fa",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## How to Use HDF5 Filters in h5py\n",
    "\n",
    "Several filters come with libhdf5, they're referred to as _built-in_. Examples: DEFLATE (known also as \"zlib\" or \"gzip\"), shuffle, scaleoffset. HDF Group also provides pre-built binary packages with a reasonable selection of other compression filters that can be installed.\n",
    "\n",
    "For Python users, there's the hdf5plugin package.\n",
    "\n",
    "Missing HDF5 filters makes the data stored using them inaccessible, thus posing a data interoperability risk."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d720e233",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The exception below usually means you are missing some filter:\n",
    "\n",
    "```\n",
    "OSError: Can't read data (can't open directory: /usr/local/hdf5/lib/plugin)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f159137c",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import hdf5plugin"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c804c3e7",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Creating a Compressed Dataset\n",
    "\n",
    "We'll create several HDF5 datasets with different compression filters using the data from the already existing dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f620a2e9",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "grp = f.create_group('compressed')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "441c22cd",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# DEFLATE compression\n",
    "grp.create_dataset('defl', data=f['dataset'][...],\n",
    "                   chunks=(20, 250),\n",
    "                   compression='gzip', compression_opts=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "445e0e03",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# ZSTD compression\n",
    "grp.create_dataset('zstd', data=f['dataset'][...],\n",
    "                   chunks=(20, 250),\n",
    "                   **hdf5plugin.Zstd(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b09cb8f1",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# ZFP fixed-accuracy lossy compression\n",
    "grp.create_dataset('zfp-fa', data=f['dataset'][...],\n",
    "                   chunks=(20, 250),\n",
    "                   **hdf5plugin.Zfp(accuracy=0.0001))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dca19cf6",
   "metadata": {},
   "outputs": [],
   "source": [
    "f.flush()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c93ce034",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "What does `**hdf5plugin.Zfp(accuracy=0.0001)` do?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ce044fb",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "hdf5plugin.Zfp(accuracy=0.0001)._kwargs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "641b68d2",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### What is the compression ratio of these different filters?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce43a0dc",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "orig = f['dataset'].id.get_storage_size()\n",
    "orig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6597d0dc",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "for dset in grp.values():\n",
    "    print(f'comp. ratio of {dset.name} = {orig / dset.id.get_storage_size() :.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cc7478c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Object References\n",
    "\n",
    "Object reference is a datatype, where its value references (points to) another HDF5 group, dataset, or committed datatype in the file. Such a value is independent of the object's current or future path name(s).\n",
    "\n",
    "The reference of an HDF5 object is obtained from its `.ref` property.\n",
    "\n",
    "We will store references to the three compressed datasets in a new dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8156a653",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "orefs = f.create_dataset(\n",
    "    'objrefs/refs', \n",
    "    data=[grp['zfp-fa'].ref, grp['defl'].ref, grp['zstd'].ref],\n",
    "    dtype=h5py.ref_dtype)\n",
    "orefs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7674af1",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "f.flush()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "267f845a",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Whenever you see an \"O\" NumPy dtype, there is something special going on behind the scenes. Let's see what this \"O\" dtype represents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffe18d45",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "orefs.dtype.metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1e50d12",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Accessing the referenced HDF5 object, _dereferencing_, uses the same syntax as for groups or datasets but with an object reference instead of a path name. Usually, the file object is used for dereferencing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96c4e5f3",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(f[orefs[1]])\n",
    "print(f'{f[orefs[-1]].name} filter id = {f[orefs[-1]].compression}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc4a935d",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Notice the last line above... h5py's filter reporting is a bit outdated and can provide misleading information for non-default filters."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e2a42b3",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Region References\n",
    "\n",
    "A value of this datatype references (points to) a region within a dataset defined by one or more hyperslabs. New region references are generated from the dataset's `regionref` property and a hyperslab selection.\n",
    "\n",
    "We're going to create region references for the compressed datasets and store them in a new dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb76d914",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "rrefs = f.create_dataset(\n",
    "    'regrefs/rrefs', \n",
    "    data=[\n",
    "        grp['zfp-fa'].regionref[:, 512:567],\n",
    "        grp['defl'].regionref[19:67, 348:412], \n",
    "        grp['zstd'].regionref[4:17, 900:]\n",
    "    ],\n",
    "    dtype=h5py.regionref_dtype)\n",
    "f.flush()\n",
    "rrefs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fcc79db1",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "rrefs.dtype.metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb275478",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Region references also serve as object references. So if using them with a group object, the return is the reference's dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "627cd0f6",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "f[rrefs[-1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f28ca1b6",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "To actually retrieve the region reference's hyperslab selection, it needs to be applied to its dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b44d967",
   "metadata": {},
   "outputs": [],
   "source": [
    "grp['zstd'][rrefs[-1]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89aa204b",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "np.array_equal(f[rrefs[-1]][rrefs[-1]], grp['zstd'][4:17, 900:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bc0e618",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.array_equal(f[rrefs[0]][rrefs[0]], grp['zfp-fa'][:, 512:567])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfcaa97e",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "f[rrefs[0]][rrefs[1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "763ed515",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Dimension Scales\n",
    "\n",
    "A dimension scale is an HDF5 dataset _attached_ to a dimension of another HDF5 dataset. This is a powerful way to record some kind of relationship between the dimension scale and its \"host\" dataset. For example, a dimension scale could hold coordinate data representing a physical quantity associated with that dimension of the \"host\" dataset.\n",
    "\n",
    "One dimension scale can be attached to more datasets, or any other dimension of the same dataset. Implementation of dimension scales uses already available HDF5 storage features so can be considered a convention.\n",
    "\n",
    "Working with dimension scales is available via the `h5py.Dataset.dims` property."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53b920a4",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We are going to: (1) create a dataset with some \"coordinate\" data; (2) make it a dimension scale; (3) attach it to all the compressed dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7ef3088",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "x = f.create_dataset('dimscale/x', data=np.arange(0, 50, .5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c663ce1",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "x.make_scale('X')\n",
    "x.is_scale"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f067ca92",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "for dset in grp.values():\n",
    "    dset.dims[0].attach_scale(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16922a89",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We can verify that a dim. scale is attached by reading some data from it via the `dims` property:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78f48f0d",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "grp['zstd'].dims[0][0].name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16a06224",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "grp['defl'].dims[0][0][5:11]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89fbc67b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a651c74",
   "metadata": {},
   "outputs": [],
   "source": [
    "!h5ls -rv {str(tgt_file)}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d407989",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Accessing HDF5 Files in the Cloud with h5py\n",
    "\n",
    "Currently two methods:\n",
    "\n",
    "* Read-only S3 (ROS3) virtual file driver (VFD). It is part of libhdf5 but must be specifically built. Good news for conda package users -- its libhdf5 package comes with this VFD. However, the PyPI version does not have it.\n",
    "* Any Python package for cloud object stores that provides a file-like object. (The fsspec package is highly recommended.)\n",
    "\n",
    "Hopefully soon -- libhdf5 REST virtual object layer (VOL) with Highly Scalable Data Service (HSDS).\n",
    "\n",
    "We're going to use the ROS3 VFD."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "942cfa3f",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "There's a significant performance difference between typical and cloud-optimized HDF5 files when in a cloud object store. We're going to use a cloud-optimized file from a NASA satellite in AWS S3 (just because it's already there).\n",
    "\n",
    "Opening this file requires a bit more arguments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1e08077",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "ros3f = h5py.File(\n",
    "    's3://hdf5.sample/data/cohdf5/PAGE8MiB_ATL03_20190928165055_00270510_004_01.h5',\n",
    "    mode='r',\n",
    "    driver='ros3', aws_region='us-west-2'.encode('ascii'),\n",
    "    page_buf_size=8 * 1024 * 1024\n",
    ")\n",
    "ros3f"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f98f0898",
   "metadata": {},
   "source": [
    "* `driver` keyword must be `ros3`\n",
    "* `page_buf_size` sets the page buffer cache at 8 MiB (only available to HDF5 files with _PAGE_ file space strategy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b32ad4d",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# List root groups members...\n",
    "list(ros3f.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c475f260",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Store all root group attributes into a dict...\n",
    "dict(ros3f.attrs.items())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d762b80",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# METADATA group's members...\n",
    "list(ros3f['METADATA'].keys())"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}