This repository holds a collection of Jupyter notebooks that demonstrate how to use Server functions, an often overlooked feature of the Hyrax data server, to perform different kinds of processing and subsetting operations on data before they leave a server.
Run these notebooks using Binder.
This project was started at part of the ESDIS PI 20.1 hackfest. If you're reading this from the NASA ECC BitBucket repository, please checkout the OPeNDAP GitHub repo of the same name since it may have more notebooks illustrating more functions.
Added 9/24/21
Subsetting level 2 satellite data generally requires regridding. However, using STARE and Hyrax server-side function, these data can be subset without first regridding them. See Subsetting Level 2 Data without Regridding for a notebook that demonstrates these new features that are the result of a NASA ACCESS grant in collaboration with Rilee, Inc., Bayesics, Inc., and UCSB.
The first notebook illustrates subsetting regularly gridded data (i.e., NASA's Processing Level 3) using Latitude and Longitude values instead of raw array indices.
This notebook reads from STARE index 'sidecar' files and plots the indices. See the README
in the stare subdirectory for information about setting up the environment. There is an
environment.ymp file but you will need to add some things not currently in conda or pip.
To develop these notebooks, we use Anaconda to establish a local package management environment. Get and install Anaconda. All our work uses Python 3. Jupyter notebooks often make use of obscure Python packages. Instead of listing them here, we'll include comments in the notebooks themselves regarding how those packages were installed in the Anaconda environment.
Anaconda modifies .bash_profile when it installs
an will set any shell to use the anaconda base environment by default,
modifying PATH to use its packages in preference to anything else (and
this breaks our builds as of 4/2020). To disable this, edit the
.bash_profile file, removing the code added by the installer at the
bottom of the file (it's easy to spot) and add a function to enable
the anaconda PATH modification. Here's an example:
function conda-on() {
source ~/opt/anaconda3/etc/profile.d/conda.sh
conda activate base
}To turn on Anaconda, just run conda-on
To use the environment.yml file to make an environment with the packages needed
to run the notebooks, use conda env create -f environment.yml. The general form of this
command is conda env create -n conda-env -f /path/to/environment.yml. The conda ...
command can also update an existing environment using the packages in a yml file with
conda env update -n conda-env -f /path/to/environment.yml.
See
Managing environments
for more information about managing environments with conda.
While an environment.yml file does not need to mention jupyter to work with Binder, to get conda
to build an environment that includes jupyter notebook support, I added ipykernel to the environment.yml
file. However, I didn't test that; it's a hunch based on my running two commands:
conda install -n opendap ipykernel# the environment name is 'opendap'python -m ipykernel install --user --name "opendap" --display-name "Python (opendap)"
I did the above but nothing seemed to work. Strange error messages from jupyter
when I try to start the notebook. I checked on the jupyter version and I saw this:
(opendap) [-bash: ~] jupyter --version
Selected Jupyter core packages...
IPython : 8.10.0
ipykernel : 6.19.2
ipywidgets : not installed
jupyter_client : 7.4.9
jupyter_core : 5.2.0
jupyter_server : not installed
jupyterlab : not installed
nbclient : not installed
nbconvert : not installed
nbformat : not installed
notebook : not installed
qtconsole : not installed
traitlets : 5.7.1So I tried installing each missing thing, and then re-checking the version. Some things,
like ipywidgets installed a bunch of the other missing pieces.
(opendap) [-bash: ~] conda install ipywidgets
(opendap) [-bash: ~] jupyter --version
Selected Jupyter core packages...
IPython : 8.10.0
ipykernel : 6.19.2
ipywidgets : 7.6.5
jupyter_client : 7.4.9
jupyter_core : 5.2.0
jupyter_server : 1.23.4
jupyterlab : not installed
nbclient : 0.5.13
nbconvert : 6.4.4
nbformat : 5.7.0
notebook : 6.5.2
qtconsole : not installed
traitlets : 5.7.1
(opendap) [-bash: ~] conda install jupyterlab
(opendap) [-bash: ~] jupyter --version
Selected Jupyter core packages...
IPython : 8.10.0
ipykernel : 6.19.2
ipywidgets : 7.6.5
jupyter_client : 7.4.9
jupyter_core : 5.2.0
jupyter_server : 1.23.4
jupyterlab : 3.5.3
nbclient : 0.5.13
nbconvert : 6.4.4
nbformat : 5.7.0
notebook : 6.5.2
qtconsole : not installed
traitlets : 5.7.1
(opendap) [-bash: ~] conda install qtconsole
(opendap) [-bash: ~]
(opendap) [-bash: ~] jupyter --version
Selected Jupyter core packages...
IPython : 8.10.0
ipykernel : 6.19.2
ipywidgets : 7.6.5
jupyter_client : 7.4.9
jupyter_core : 5.2.0
jupyter_server : 1.23.4
jupyterlab : 3.5.3
nbclient : 0.5.13
nbconvert : 6.4.4
nbformat : 5.7.0
notebook : 6.5.2
qtconsole : 5.4.0
traitlets : 5.7.1
(opendap) [-bash: ~/OPeNDAP/hyrax/notebooks] And then everything worked!
Once you have gotten everything installed (see previous section) you run the notebooks from the command line like this:
jupyter notebook Geospatial_subsetting_using_server_functions.ipynb
jupyter notebook tutorials/pydap_dap2_basic.ipynbHere are resources we've found useful in building these notebooks:
- Advanced Jupyter Notebook Tricks — Part I: Various magic codes like %time
- Requests: An HTTP/S request package
- Basemap: A tool for drawing georeferenced maps.
- PyDAP: A flexible tool for accessing data from OPeNDAP servers.
- matplotlib: The plotter of choice for many.
- matplotlib basemap: Soon to be deprecated for several years now...
- Cartopy in Pypi
- Cartopy: The successor to Basemap.
- Cartopy Basics
- Cartopy more...
- Markdown for notebooks: Just in case you need a primer..
- Binder Documentation is at the bottom of the page.
- H5Py
Collections of Jupyter notebooks that also demonstrate OPeNDAP. There are many others, but these will get you started with access patterns that don't use server functions:
- CloudyDAP notebooks: Written by Joe Lee from The HDF Group.
- PyDAP Examples: Written by Beto DiAlmeida, author of PyDAP.