Accessing (scientific) data on the internet without having to download it

[amotl]

Hi there,

just recently, we became aware about different technologies which delegate data processing to server side components instead of having to download and process individual resources completely. This topic is dedicated to collect information about different implementations and discuss eventual evaluations.

With kind regards,
Andreas.

[amotl]

Looking at the Birdhouse software ecosystem

At #282 (reply in thread), we found interesting pieces of software centered around the OGC Web Processing Service (WPS) Interface Standard, mainly conceived by the @geopython, @bird-house, @roocs and @Ouranosinc communities.

About

Birdhouse tools enable you to build your own customised Web Processing Service application in support of web-based geospatial (climate) data analysis.

-- https://bird-house.github.io/

Details

The idea is that instead of downloading large volumes of data locally and then analyzing it, part of the analysis can be done remotely by servers, close to the source data. Instead of users sending a plain download request, users send a request for pre-processed data. Work with low-added value can be delegated to a server, and the real research is performed on reduced datasets on local machines.

-- https://pavics-sdi.readthedocs.io/projects/raven/

References

• https://github.com/geopython/pywps
• https://github.com/bird-house/birdy
• https://github.com/bird-house/flyingpigeon
• https://github.com/bird-house/finch
• https://github.com/roocs/rooki
• https://github.com/roocs/rook
• https://github.com/roocs/daops
• https://github.com/Ouranosinc/raven

With kind regards,
Andreas.

[amotl]

@Zeitsperre also made us aware about what they are doing over at @Ouranosinc within #282 (reply in thread):

Raven is part of the Birdhouse project, which is a series of WPS services that aid in performing climate, hydrology, terrain, and EO-like workflows using remote data/servers.

[amotl]

Looking at DAP4

Upstream says that the Data Access Protocol (DAP) is widely used, especially in Earth science. The protocol is layered on HTTP and its current specification is DAP4. DAP is intended to enable remote, selective data-retrieval as an easily invoked Web service.

OPeNDAP is the Open-source Project for a Network Data Access Protocol. OPeNDAP, Inc. develops and maintains zero-cost (reference) implementations of the DAP protocol in both server-side and client-side software.

• https://docs.opendap.org/index.php/DAP4_Specification
• https://www.opendap.org/
• https://en.wikipedia.org/wiki/OPeNDAP

@mhaberler made us aware about pydap. It is:

A Python library implementing the Data Access Protocol (DAP, aka OPeNDAP or DODS).

pydap is an implementation of the OPeNDAP/DODS protocol, written from scratch in pure python. You can use pydap to access scientific data on the internet without having to download it; instead, you work with special array and iterable objects that download data on-the-fly as necessary, saving bandwidth and time.

[mhaberler]

I have not much practical experience with pydap yet as far as status, functionality and stability goes

the docs link is broken, and there is a valid PR sitting there for 2 years, strange

have a look at the network graph: https://github.com/pydap/pydap/network and in particular the https://github.com/betodealmeida/pydap/network fork

I'll see what I can find out as to where things are happening in pydap

[amotl]

Thanks Michael.

@MBARIMike also pointed out at pydap/pydap#167 (comment):

It's been over a year since there's been a new release of Pydap. Perhaps it's time to push a new one out to PyPI and Conda?

Also, @jerabaul29 pointed out at pydap/pydap#212:

It looks like the last PyDap release was in 2017.

[amotl]

Hi Michael,

I'll see what I can find out as to where things are happening in pydap.

I've just made my way through installing and operating pydap and contributed a wrap-up at pydap/pydap#212 (comment). It includes your suggestion to use Gunicorn 19.x instead of the most recent 20.x release.

With kind regards,
Andreas.

[mhaberler]

I've set up a pyndap sandbox server at https://pydap.mah.priv.at/

Here's a remote access example:

import xarray as xr
ds_url = 'https://pydap.mah.priv.at/all.nc'
ds =  xr.open_dataset(ds_url)
print(ds)

Test drive:

(dev-3.8) mah@next:~/src/pydap-tests$ python xarr.py
<xarray.Dataset>
Dimensions:    (bnds: 2, lat: 76, lon: 115, plev: 1, time: 1)
Coordinates:
  * time       (time) datetime64[ns] 2020-12-15T14:00:00
  * lon        (lon) float64 14.5 14.52 14.54 14.56 ... 16.72 16.74 16.76 16.78
  * lat        (lat) float64 46.0 46.02 46.04 46.06 ... 47.44 47.46 47.48 47.5
  * plev       (plev) float64 6e+04
  * bnds       (bnds) int32 0 1
Data variables:
    plev_bnds  (plev, bnds) float64 ...
    CLCM       (time, plev, lat, lon) float32 ...
Attributes:
    CDI:            Climate Data Interface version 1.9.6 (http://mpimet.mpg.d...
    Conventions:    CF-1.6
    history:        Wed Dec 16 23:37:32 2020: cdo -f nc copy /var/www/static....
    institution:    Deutscher Wetterdienst
    CDO:            Climate Data Operators version 1.9.6 (http://mpimet.mpg.d...
    Last-modified:  Sat, 19 Dec 2020 10:50:53 -0000

code updated as per @amotl 's suggestion

[mhaberler]

I asked and was advised caution, for instance this issue

Juan recommended to use https://github.com/OPENDAP/hyrax-docker instead which is at least a way to encapsulate Java-related insanities

[amotl]

Thanks a bunch for these pointers.

When giving Hyrax a test drive, I also found that the most recent official Pydap release from PyPI will not be able to access resources because Hyrax serves them with gzip compression, which actually makes sense when considering efficiency.

However, this issue has been fixed already through pydap/pydap@84e85db (see pydap/pydap#167) and can be mitigated by installing the most recent version of Pydap directly from the Git repository.

I have added a wrap-up to this at pydap/pydap#212 (comment).

Thanks @laliberte and @MBARIMike.