Merge pull request #450 from dbekaert/dev

Release: v0.3.0

.circleci/config.yml

@@ -43,7 +43,7 @@ jobs:
             eval "$($HOME/bin/micromamba shell hook -s posix)"
             micromamba activate RAiDER
             python -m pip install .
-            python -c "import RAiDER; from RAiDER.delay import main"
+            python -c "import RAiDER; from RAiDER.delay import tropo_delay"
             python -c "import RAiDER; from RAiDER.interpolator import interp_along_axis"
       - run:
           name: Run unit tests

CHANGELOG.md

@@ -6,6 +6,25 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [PEP 440](https://www.python.org/dev/peps/pep-0440/) 
 and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.3.0]
+RAiDER package was refactored to expose the main functionality as a Python library, including the `prepareWeatherModel` 
+and `tropo_delay` functions, as well as anciliarry functions needed for defining AOIs, look vectors, etc.
+
+### New/Updated Features
++ Python library access to main functions for accessing weather model data and calculating delays
++ Slant delay calculation through projection is supported for cubes with orbit files
++ Upgrade dem-stitcher to [`>=2.3.1`](https://github.com/ACCESS-Cloud-Based-InSAR/dem-stitcher/blob/dev/CHANGELOG.md#231) so that the updated urls for the GLO-30 DEM are used.
++ `raider.py ++calcDelaysGUNW GUNWFILE` is enabled as a placeholder only.
++ Upgraded ISCE3 to `>=v0.9.0` to fix a conda build issue as described in [#425](https://github.com/dbekaert/RAiDER/issues/425)
++ Allow user to specify --download_only or download_only=True in the configure file
++ Added documentation for the Python library interface. 
++ Added some unit tests. 
++ Fixed some bugs and tweaked the CLI.
++ Added unit tests, docstrings, initial API reference
++ __main__ file to allow calls to different functionality. `raider.py ++process downloadGNSS ...` can now perform the functionality of `raiderDownloadGNSS.py ...
+
+
+
 ## [0.2.0]
 
 RAiDER package was refactored to use a configure file (yaml) to parse parameters. In addition, ocker container images

README.md

@@ -1,4 +1,5 @@
 # RAiDER
+
 Raytracing Atmospheric Delay Estimation for RADAR
 
 [![Language](https://img.shields.io/badge/python-3.7%2B-blue.svg)](https://www.python.org/)
@@ -13,15 +14,18 @@ Copyright (c) 2019-2022, California Institute of Technology ("Caltech"). All rig
 THIS IS RESEARCH CODE PROVIDED TO YOU "AS IS" WITH NO WARRANTIES OF CORRECTNESS. USE AT YOUR OWN RISK.
 
 ## Contents
-- [1. Getting Started](#1-getting-started)
-  - [Installing With Conda](#installing-with-conda)
-  - [Using the Docker Image](#using-the-docker-image)
-  - [Installing from Source](#installing-from-source)
-- [2. Setup of third party weather model access](#2-setup-of-third-party-weather-model-access)
-- [3. Running RAiDER and Documentation](#3-running-raider-and-documentation)
-- [4. Citing](#4-citation)
-- [5. Contributors](#5-contributors)
+
+1. [Getting Started](#1-getting-started)
+    - [Installing With Conda](#installing-with-conda)
+    - [Using the Docker Image](#using-the-docker-image)
+    - [Installing from Source](#installing-from-source)
+2. [Setup of third party weather model access](#2-setup-of-third-party-weather-model-access)
+3. [Running RAiDER and Documentation](#3-running-raider-and-documentation)
+4. [Citing](#4-citation)
+5. [Development](#5-development)
+    - [Contributors](#contributors) 
 ------
+
 ## 1. Getting Started
 
 RAiDER has been tested on the following systems:
@@ -31,6 +35,7 @@ RAiDER has been tested on the following systems:
 RAiDER does **not** currently run on arm64 processors on Mac. We will update this note once the build becomes available. 
 
 ### Installing With Conda
+
 RAiDER is available on [conda-forge](https://anaconda.org/conda-forge/raider). __[Conda](https://docs.conda.io/en/latest/index.html)__ is a cross-platform way to use Python that allows you to setup and use "virtual environments." These can help to keep dependencies for different sets of code separate. We recommend using [Miniforge](https://github.com/conda-forge/miniforge), a conda environment manager that uses conda-forge as its default code repo. Alternatively,see __[here](https://docs.anaconda.com/anaconda/install/)__ for help installing Anaconda and __[here](https://docs.conda.io/en/latest/miniconda.html)__ for installing Miniconda.
 
 Installing RAiDER:
@@ -40,6 +45,7 @@ conda activate RAiDER
 ```
 
 ### Using the Docker image
+
 RAiDER provides a [docker container image](https://docs.docker.com/get-started/) with all the necessary dependencies pre-installed. To get the latest released version: 
 ```
 docker pull ghcr.io/dbekaert/raider:latest
@@ -64,32 +70,43 @@ cd work
 ```
 For more docker run options, see: <https://docs.docker.com/engine/reference/run/>.
 
-### Installing from source
-You can also install RAiDER directly from source. Doing so is recommended for those who would like to [contribute to the source code](https://github.com/dbekaert/RAiDER/blob/dev/CONTRIBUTING.md), which we heartily encourage! For more details on installing from source see [here](https://github.com/dbekaert/RAiDER/blob/dev/Installing_from_source.md).
-```
-git clone https://github.com/dbekaert/RAiDER.git
-cd RAiDER
-conda create -f environment.yml
-conda activate RAiDER
-python -m pip install -e .
-```
+
 ------
 ## 2. Setup of third party weather model access
-RAiDER has the ability to download weather models from third-parties; some of which require license agreements. See [here](WeatherModels.md) for details.
+
+RAiDER has the ability to download weather models from third-parties; some of which require license agreements. See [here](https://github.com/dbekaert/RAiDER/blob/dev/docs/WeatherModels.md) for details.
 
 ------
 ## 3. Running RAiDER and Documentation
+
 For detailed documentation, examples, and Jupyter notebooks see the [RAiDER-docs repository](https://github.com/dbekaert/RAiDER-docs).
-We welcome contributions of other examples on how to leverage the RAiDER  (see [here](https://github.com/dbekaert/RAiDER/blob/master/CONTRIBUTING.md) for instructions).
+We welcome contributions of other examples on how to leverage the RAiDER  (see [here](https://github.com/dbekaert/RAiDER/blob/dev/CONTRIBUTING.md) for instructions).
 ``` raiderDelay.py -h ``` provides a help menu and list of example commands to get started.
-The RAiDER scripts are highly modulized in Python and allows for building your own processing workflow.
+The RAiDER scripts are highly modularized in Python and allows for building your own processing workflow.
 
 ------
 ## 4. Citation
 TODO
 
 ------
-## 5. Contributors
+## 5. Development
+
+Contributions are welcome and heartily encourage! See our [contributing guide](https://github.com/dbekaert/RAiDER/blob/dev/CONTRIBUTING.md).
+
+### Development install
+For development, we recommend installing directly from source.
+```
+git clone https://github.com/dbekaert/RAiDER.git
+cd RAiDER
+conda create -f environment.yml
+conda activate RAiDER
+python -m pip install -e .
+```
+For more details on installing from source see [here](https://github.com/dbekaert/RAiDER/blob/dev/docs/Installing_from_source.md).
+
+------
+### Contributors
+
 * David Bekaert
 * Jeremy Maurer
 * Raymond Hogenson

WeatherModels.md → docs/WeatherModels.md

@@ -1,30 +1,28 @@
 # Accessing weather model data
-RAiDER has built-in support for a number of different weather models.  RAiDER provides all of the interfacing to data servers required to access data for the different weather models, although some weather models require a license agreement and accounts to be set-up. Instructions for accessing data, including license-limited data, are provided below. It is the user's responsibility to accept license agreements for whatever model is desired. 
+RAiDER has built-in support for a number of different weather models.  RAiDER provides all the interfacing to data servers required to access data for the different weather models, although some weather models require a license agreement and accounts to be set-up. Instructions for accessing data, including license-limited data, are provided below. It is the user's responsibility to accept license agreements for whatever model is desired. 
 
 In addition, RAiDER provides functionality for adding additional weather models. See the [RAiDER-docs repository](https://github.com/dbekaert/RAiDER-docs) page on how to do this. We would love to expand the suite of supported models, and welcome any contributions. Please see the contributing guidelines or reach out through an issue ticket for help. 
 
 ------
 
-## Contents
+## 1. Usage
 
-1. [NOAA weather models (HRRR)](#noaa-weather-models-(hrrr))
-2. [ECMWF weather models (ERA5, ERA5T, ERAI, HRES)](#ecmwf-weather-models-(era5,-era5t,-erai,-hres))
-3. [NASA weather models (GMAO, MERRA2)](#nasa-weather-models-(gmao,-merra2))
+::: RAiDER.processWM.prepareWeatherModel
+    options:
+      show_root_heading: true
+      heading_level: 3
 
-#Potential download failure:#
+### Potential download failure
 ERA-5/ERA-I products require access to the ESA Copernicus servers. GMAO and MERRA-2 products require access to the NASA Earthdata servers. If you are unable to download products, ensure that you have registered and have downloaded the public API key, and accepted/added the license/application for type of product you wish to download as detailed below. 
 
 ------
 
-## 1. NOAA weather models (HRRR)
-High-resolution rapid refresh (HRRR) weather model data products are generated by __[NOAA](https://rapidrefresh.noaa.gov/hrrr/)__ for the coninental US (CONUS) but not archived beyond three days. However a public __[archive](home.chpc.utah.edu/~u0553130/Brian_Blaylock/hrrr_FAQ.html)__ is available at the University of Utah. This archive does not require a license agreement. This model has the highest spatial resolution available in RAiDER, with a horizontal grid spacing of about 3 km, and is provided in a Lambert conformal conic projection. 
-
-[return to table of content](#contents)
-
+## 2. NOAA weather models (HRRR)
+High-resolution rapid refresh (HRRR) weather model data products are generated by __[NOAA](https://rapidrefresh.noaa.gov/hrrr/)__ for the coninental US (CONUS) but not archived beyond three days. However, a public __[archive](home.chpc.utah.edu/~u0553130/Brian_Blaylock/hrrr_FAQ.html)__ is available at the University of Utah. This archive does not require a license agreement. This model has the highest spatial resolution available in RAiDER, with a horizontal grid spacing of about 3 km, and is provided in a Lambert conformal conic projection. 
 
 ------
 
-## 2. ECMWF weather models (ERA5, ERA5T, ERAI, HRES)
+## 3. ECMWF weather models (ERA5, ERA5T, ERAI, HRES)
 The Copernicus Climate Data Store (CDS) provides access to the European Centre for Medium-Range Weather Forecasts (__[ECMWF](https://www.ecmwf.int/)__)  provides a number of different weather models, including ERA5 and ERA5T reanalysis models.  
 
 The ECMWF provides access to both reanalysis and real-time prediction models. You can read more information about their reanalysis models __[here](https://www.ecmwf.int/en/research/climate-reanalysis)__ and real-time model __[here](https://www.ecmwf.int/en/forecasts/datasets/catalogue-ecmwf-real-time-products)__. ECMWF models are global, with horizontal resolution of about 30 km for ERA-I, ERA-5, and ERA-5T, and 6 km for Hi-RES. All of these models come in a global projection (EPSG 4326, WGS-84). 
@@ -34,41 +32,47 @@ The ECMWF provides access to both reanalysis and real-time prediction models. Yo
 2. Confirm your email, etc. 
 3. Install the public API key and client as instructed __[here](https://cds.climate.copernicus.eu/api-how-to)__:
 
-   a. Copy the URL and API key from the webpage into a file in your home directory name ~/.cdsapirc 
-         url: https://cds.climate.copernicus.eu/api/v2
-         key: your_key_here
-      __**Note**: the key represents the API key obtained upon the registration of CDS API, and should be replaced with the user's own information.__
-
-   b. Install the CDS API using pip: 
-         pip install cdsapi
-   ___**Note**: this step has been included in the conda install of RAiDER, thus can be omitted if one uses the recommended conda install of RAiDER___
+    a. Copy the URL and API key from the webpage into a file in your home directory name `~/.cdsapirc`
+
+        url: https://cds.climate.copernicus.eu/api/v2
+        key: your_key_here
+
+    **Note**: the key represents the API key obtained upon the registration of CDS API, and should be replaced with the user's own information.
+
+    b. Install the CDS API using pip
+
+        pip install cdsapi
+
+    **Note**: this step has been included in the conda install of RAiDER, thus can be omitted if one uses the recommended conda install of RAiDER
+
 4. You must accept the [license](https://cds.climate.copernicus.eu/cdsapp/#!/terms/licence-to-use-copernicus-products) for each product you wish to download.
 
 
 ### Accessing ERAI, HRES
 
 ECMWF requires a license agreement to be able to access, download, and use their products. Instructions for completing this process is below. 
 
-1. Create an account on the ECMWF servers __[here](https://accounts.ecmwf.int/auth/realms/ecmwf/protocol/openid-connect/auth?response_type=code&scope=openid%20email&client_id=apache-www&state=sBYlpcTRPhat8d6uuM9swLCxuP8&redirect_uri=https%3A%2F%2Fwww.ecmwf.int%2Foidc.cgi&nonce=RyEzBUy4m6oo_HxRQEmJxbc5jrKY4KFZd1Usgi8cpnM)__. The ERA-I model is open-access, while HRES requires a special liscence agreement.
+1. Create an account on the ECMWF servers __[here](https://accounts.ecmwf.int/auth/realms/ecmwf/protocol/openid-connect/auth?response_type=code&scope=openid%20email&client_id=apache-www&state=sBYlpcTRPhat8d6uuM9swLCxuP8&redirect_uri=https%3A%2F%2Fwww.ecmwf.int%2Foidc.cgi&nonce=RyEzBUy4m6oo_HxRQEmJxbc5jrKY4KFZd1Usgi8cpnM)__. The ERA-I model is open-access, while HRES requires a special licence agreement.
 2. Confirm your email, etc. 
 3. Install the public API key and client as instructed __[here](https://confluence.ecmwf.int/display/WEBAPI/Access+ECMWF+Public+Datasets#AccessECMWFPublicDatasets-key)__: 
 
-    a. Copy the URL and API key from the webpage into a file in your home directory name ~/.ecmwfapirc 
-    {
-        "url"   : "https://api.ecmwf.int/v1",
-        "key"   : your key here,
-        "email" : your email here
-    }
+    a. Copy the URL and API key from the webpage into a file in your home directory name `~/.ecmwfapirc`
+
+        {
+            "url"   : "https://api.ecmwf.int/v1",
+            "key"   : your key here,
+            "email" : your email here
+        }
 
-    __**Note**: the email that is used to register the user account, and the key represents the API key obtained upon the registration of ECMWF API, and should be replaced with the user's own information.__
+    **Note**: the email that is used to register the user account, and the key represents the API key obtained upon the registration of ECMWF API, and should be replaced with the user's own information.
 
-    b. Install the ECMWF API using pip: 
-        ```pip install ecmwf-api-client```
-    ___**Note**: this step has been included in the conda install of RAiDER, thus can be omitted if one uses the recommended conda install of RAiDER___
+    b. Install the ECMWF API using pip:
 
-[return to table of content](#contents)
+        pip install ecmwf-api-client`
 
-### 3. NASA weather models (GMAO, MERRA2)
+    **Note**: this step has been included in the conda install of RAiDER, thus can be omitted if one uses the recommended conda install of RAiDER
+
+## 4. NASA weather models (GMAO, MERRA2)
 
 1. The Global Modeling and Assimilation Office (__[GMAO](https://www.nccs.nasa.gov/services/data-collections/coupled-products/geos5-forecast#:~:text=The%20Global%20Modeling%20and%20Assimilation,near%2Dreal%2Dtime%20production.)__) at NASA generates reanalysis weather models. GMAO products can also be accessed without a license agreement through the pyDAP interface implemented in RAiDER. GMAO has a horizontal grid spacing of approximately 33 km, and its projection is EPSG code 4326 (WGS-84). 
 
@@ -85,17 +89,14 @@ Reference: __[The Modern-Era Retrospective Analysis for Research and Application
                  login <USERNAME>
                  password <PASSWORD>
 
-   __**Note**: the username and password represent the user's username and password.__
+    **Note**: the username and password represent the user's username and password.
 
 4. Add the application `NASA GESDISC DATA ARCHIVE` by clicking on the `Applications->Authorized Apps` on the menu after logging into your Earthdata profile, and then scrolling down to the application `NASA GESDISC DATA ARCHIVE` to approve it. _This seems not required for GMAO for now, but recommended to do so for all OpenDAP-based weather models._
 5. Install the OpenDAP using pip: 
 
          pip install pydap==3.2.1
       
 
-   ___**Note**: this step has been included in the conda install of RAiDER, thus can be omitted if one uses the recommended conda install of RAiDER___
+    **Note**: this step has been included in the conda install of RAiDER, thus can be omitted if one uses the recommended conda install of RAiDER
 
-   ___**Note**: PyDAP v3.2.1 is required for now (thus specified in the above pip install command) because the latest v3.2.2 (as of now) has a known [bug](https://colab.research.google.com/drive/1f_ss1Oa3VzgAOd_p8sgekdnLVE5NW6s5) in accessing and slicing the GMAO data. This bug is expected to be fixed in newer versions of PyDAP.___
-
-[return to table of content](#contents)
-
+    **Note**: PyDAP v3.2.1 is required for now (thus specified in the above pip install command) because the latest v3.2.2 (as of now) has a known [bug](https://colab.research.google.com/drive/1f_ss1Oa3VzgAOd_p8sgekdnLVE5NW6s5) in accessing and slicing the GMAO data. This bug is expected to be fixed in newer versions of PyDAP.

docs/index.md

@@ -0,0 +1 @@
+../README.md

docs/macros.py

@@ -0,0 +1,17 @@
+import requests
+
+import RAiDER
+
+
+def define_env(env):
+    """Macros Hook"""
+
+    @env.macro
+    def raider_version():
+        return RAiDER.__version__
+
+    @env.macro
+    def get_content(url):
+        response = requests.get(url)
+        response.raise_for_status()
+        return response.content.decode()

docs/reference.md

@@ -0,0 +1,5 @@
+# RAiDER *v{{ raider_version() }}* API Reference
+
+::: RAiDER
+    options:
+      show_submodules: true

docs/reference_td.md

@@ -0,0 +1,3 @@
+***`tropo_delay`***
+
+::: RAiDER.delay

docs/tutorials.md

@@ -0,0 +1,3 @@
+### Tutorials
+
+{{ get_content('https://raw.githubusercontent.com/dbekaert/RAiDER-docs/main/README.md') }}

environment.yml

@@ -17,12 +17,12 @@ dependencies:
  - cxx-compiler
  - cython
  - dask
- - dem_stitcher>=2.3.0
+ - dem_stitcher>=2.3.1
  - ecmwf-api-client
  - h5py
  - herbie-data
  - h5netcdf
- - isce3==0.8.0=*_2
+ - isce3>=0.9.0
  - lxml
  - matplotlib
  - netcdf4
@@ -31,11 +31,14 @@ dependencies:
  - progressbar
  - pydap>3.2.2
  - pyproj>=2.2.0
+ - pyyaml
  - rasterio>=1.3.0
+ - requests
  - s3fs
  - scipy
  - shapely
  - sysroot_linux-64
+ - tqdm
  - xarray
  # For packaging and testing
  - autopep8
@@ -44,7 +47,13 @@ dependencies:
  - pytest-timeout
  - pytest-console-scripts
  - setuptools_scm >=6.2
-
+ # For docs website
+ - mkdocs
+ - mkdocstrings
+ - mkdocstrings-python
+ - mkdocs-macros-plugin
+ - mkdocs-material
+ - mkdocs-material-extensions
  # For RAiDER-docs
  - jupyterlab
  - jupyter_contrib_nbextensions