Skip to content

Commit

Permalink
Merge pull request #461 from dbekaert/dev
Browse files Browse the repository at this point in the history
Release v0.4.0
  • Loading branch information
jlmaurer authored Dec 24, 2022
2 parents a2c6cb6 + 06fffb6 commit 2c450e0
Show file tree
Hide file tree
Showing 30 changed files with 846 additions and 308 deletions.
68 changes: 50 additions & 18 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,46 @@
All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [PEP 440](https://www.python.org/dev/peps/pep-0440/)
and this project adheres to [PEP 440](https://www.python.org/dev/peps/pep-0440/)
and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.4.0]

Adding of new GUNW support to RAiDER. This is an interface delivery allowing for subsequent integration into HYP3 (input/output parsing is not expected to change; computed data is not yet verified).

### New/Updated Features
+ Working GUNW entry point in workflow for raider.py
+ Ability to parse a GUNW to workflows from which all required RAiDER information is extracted (e.g. dates, UTC, orbit, bbox, look direction, wavelength) with an option to specify weather model (those already supported by RAiDER) and ability to squeeze in the derived output into the original GUNW product.
+ Delays for GUNW are calculated in RAiDER using the ray-tracing option specifying bbox (GUNW driven), a hardcoded lateral posting (0.05º for HRRR and 0.1º for others), fixed vertical height levels, using an different orbit file for secondary and master.
- The hard-coded heights and posting will be refined per model and to ensure stitching abilities in ARIA-tools.
- The orbit should be refined to not change between secondary and reference to avoid issues. See https://github.com/dbekaert/RAiDER/discussions/435#discussioncomment-4392665
+ Bug fix for raider.py "date" input argument when multiple dates are requested (i.e. support of requesting two dates or two dates with a sampling).
+ Add unit test for date input argument checking (single day, two dates, two dates with samples)
+ Write the diagnostic weather model files to the 'output_directory' rather than PWD
+ Fix for incorrectly written hard-cored projection embedded in the computed output data
+ Allow for multiple orbits files/dates to be used for slant:projection
+ correctly pass llh to lla_to_ecef function for slant:projection
++ verified this doesnt change anything
+ removed deprecated ray projection functionality
+ added 1º buffer for zenith and projected (already done for ray tracing)
+ differential delay is rounded to model-dependent nearest hour
+ version 1c hardcoded into the updated GUNW

### Added dependencies for:
+ sentinelof: used to fetch the orbit for GUNW
+ rioxarray: used for reading rasters with xarray

### Not implemented / supported in this release### Not implemented / supported in this release
+ no temporal interpolation
+ no refined model specific hardcoded spacing and heights
+ no ability for single orbit Interferometric calculation
+ no verification of results

## [0.3.1]
Fixes some missing imports and typing statements

## [0.3.0]
RAiDER package was refactored to expose the main functionality as a Python library, including the `prepareWeatherModel`
RAiDER package was refactored to expose the main functionality as a Python library, including the `prepareWeatherModel`
and `tropo_delay` functions, as well as anciliarry functions needed for defining AOIs, look vectors, etc.

### New/Updated Features
Expand All @@ -20,8 +52,8 @@ and `tropo_delay` functions, as well as anciliarry functions needed for defining
+ `raider.py ++calcDelaysGUNW GUNWFILE` is enabled as a placeholder only.
+ Upgraded ISCE3 to `>=v0.9.0` to fix a conda build issue as described in [#425](https://github.com/dbekaert/RAiDER/issues/425)
+ Allow user to specify --download_only or download_only=True in the configure file
+ Added documentation for the Python library interface.
+ Added some unit tests.
+ Added documentation for the Python library interface.
+ Added some unit tests.
+ Fixed some bugs and tweaked the CLI.
+ Added unit tests, docstrings, initial API reference
+ __main__ file to allow calls to different functionality. `raider.py ++process downloadGNSS ...` can now perform the functionality of `raiderDownloadGNSS.py ...
Expand All @@ -32,17 +64,17 @@ and `tropo_delay` functions, as well as anciliarry functions needed for defining

RAiDER package was refactored to use a configure file (yaml) to parse parameters. In addition, ocker container images
are provided with all the necessary dependencies pre-installed. Various models were tested for consistency
(others were disabled for the time being) with propagation delays computation support for zenith, and slant through
ray-tracing using orbit file. Modules were restructured for computational performance improvement.
(others were disabled for the time being) with propagation delays computation support for zenith, and slant through
ray-tracing using orbit file. Modules were restructured for computational performance improvement.

This release is the initial release to the NISAR ADT for supporting the generation of the stratospheric ancillary
correction using HRES model.
correction using HRES model.

### New/Updated Features
+ Supported for models (verified)
- ECWMF: HRES, ERA5, and ERA5T access verified and tested
- NASA GSFC: GMAO access fix for PYDAP changes
- NOAA: HRRR access through AWS S3 bucket (using [Herbie](https://github.com/blaylockbk/Herbie)) and correction on
- NOAA: HRRR access through AWS S3 bucket (using [Herbie](https://github.com/blaylockbk/Herbie)) and correction on
incorrect loading of the pressure.
- NCMR: not tested
- Other models are currently disabled (MERRA-2, WRF, ERA-I)
Expand All @@ -52,22 +84,22 @@ correction using HRES model.
+ Refactoring and computational improvements for delay computation
- Better organized for individual function calls
+ When using a pre-existing weather model file, raise a warning instead of an error when the existing weather model
doesn't cover the entire AOI.
doesn't cover the entire AOI.
+ Support for delay outputs at
- 2D coordinate list (x,y) for a user-defined height or at topographic height
- 2D grid at user-specified output sampling (x,y) and coordinate projection for a user-defined height or by default
at topographic height
- 3D cube at weather model grid notes or at user-specified output sampling (x,y,z) and coordinate projection
at topographic height
- 3D cube at weather model grid notes or at user-specified output sampling (x,y,z) and coordinate projection
+ Docker container images are provided with all the necessary dependencies pre-installed. See: <https://github.com/dbekaert/RAiDER/#using-the-docker-image>
+ CLI has changed from `raiderDelay.py` to `raider.py` with options provided through configure file (yaml format).
+ CLI has changed from `raiderDelay.py` to `raider.py` with options provided through configure file (yaml format).
+ Unit testing for slant ray-tracing
+ RAiDER-Docs documentation updated for changes in RAiDER package

### Not implemented / supported in this release
+ Custom DEMs
+ Custom DEMs
+ Pypi (`pip install` capability)
+ Slant delays without an orbit file (e.g. own 2D or 3D LOS files)
+ Conventional slant delays with projection from zenith (e.g. orbit, 2D or 3D LOS files)
+ Slant delays without an orbit file (e.g. own 2D or 3D LOS files)
+ Conventional slant delays with projection from zenith (e.g. orbit, 2D or 3D LOS files)
+ GUNW product input/output
+ Complete unit test coverage
+ Detailed API documentation
Expand Down Expand Up @@ -104,7 +136,7 @@ Nate Kean
### RAiDER pre-release in support of NISAR troposphere working group
This release includes features in support of the NISAR tropospheric working group recommendations in June 2021. Salient
features include full capabilities for statistical analysis of GNSS ZTD and comparison with weather model ZTD, addition
of the NCMR weather model from ISRO, and unit test suite update.
of the NCMR weather model from ISRO, and unit test suite update.

### Pre-release
- GMAO, MERRA-2, ECMWF - ERA-5, ERA-5T, HRES, HRRR, and NCMR all working models
Expand All @@ -126,7 +158,7 @@ Brett Buzzanga
### RAiDER pre-release in support of AGU work
This release includes the features in support of the team's 2020 AGU presentation. Salient features include model
download, GNSS ZTD download, ZTD delay calculation at GNSS locations, statistical analysis of the GNSS and model ZTD.
Slant delay and large scale processing in ongoing development.
Slant delay and large scale processing in ongoing development.

### Pre-release
- Concurrent download option (tested for GMAO, HRES, ERA5)
Expand All @@ -149,7 +181,7 @@ Brett Buzzanga

### First RAiDER pre-release
Predominant to be used for model download in support of upcoming statistical analysis for NISAR tropospheric noise working group.
Delay calculation and stats class in ongoing development.
Delay calculation and stats class in ongoing development.

### Pre-release
- Download support for HRRR, GMAO, HRES, ERA5, ERAT, MERRA2
Expand Down
2 changes: 2 additions & 0 deletions environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ dependencies:
- pyproj>=2.2.0
- pyyaml
- rasterio>=1.3.0
- rioxarray
- requests
- s3fs
- scipy
Expand All @@ -54,6 +55,7 @@ dependencies:
- mkdocs-macros-plugin
- mkdocs-material
- mkdocs-material-extensions
- sentineleof
# For RAiDER-docs
- jupyterlab
- jupyter_contrib_nbextensions
Expand Down
Binary file not shown.
Binary file removed test/scenario_1/HRRR_tropo_20200101T120000_ztd.nc
Binary file not shown.
Binary file not shown.
2 changes: 2 additions & 0 deletions test/scenario_1/raider_example_1.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,7 @@
height_group:
height_levels: 0 50 100 500 1000 # Return only these specific height levels
los_group: # absent other options ZTD is calculated
runtime_group:
output_directory: test/scenario_1


1 change: 1 addition & 0 deletions test/scenario_3/raider_example_3.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,4 @@ los_group:
runtime_group:
output_projection: 4326
cube_spacing_in_m: 5000.0
output_directory: test/scenario_3
1 change: 1 addition & 0 deletions test/scenario_3/raider_example_3_proj.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,4 @@ los_group:
runtime_group:
output_projection: 4326
cube_spacing_in_m: 5000.0
output_directory: test/scenario_3
64 changes: 64 additions & 0 deletions test/test_GUNW.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
import os
import shutil
import numpy as np
import pytest
import shutil
import subprocess
import xarray as xr
import rasterio as rio

from test import TEST_DIR


def test_GUNW():
## eventually to be implemented
# home = os.path.expanduser('~')
# netrc = os.path.join(home, '.netrc')
#
# ## make netrc
# if not os.path.exists(netrc):
# name, passw = os.getenv('URSname'), os.getenv('URSpass')
# cmd = f'echo "machine urs.earthdata.nasa.gov login {name} password {passw}" > ~/.netrc'
# subprocess.run(cmd.split())
#
# cmd = f'chmod 600 {netrc}'
# subprocess.run(cmd.split())
#
SCENARIO_DIR = os.path.join(TEST_DIR, "GUNW")
os.makedirs(SCENARIO_DIR, exist_ok=True)
GUNW = 'S1-GUNW-D-R-071-tops-20200130_20200124-135156-34956N_32979N-PP-913f-v2_0_4.nc'
orig_GUNW = os.path.join(TEST_DIR, GUNW)
updated_GUNW = os.path.join(SCENARIO_DIR, GUNW)
shutil.copy(orig_GUNW, updated_GUNW)


# cmd = f'wget https://grfn.asf.alaska.edu/door/download/{GUNW}'
# proc = subprocess.run(cmd.split(), stdout=subprocess.PIPE, universal_newlines=True)
# assert np.isclose(proc.returncode, 0)

cmd = f'raider.py ++process calcDelaysGUNW {updated_GUNW} -m GMAO -o {SCENARIO_DIR}'
proc = subprocess.run(cmd.split(), stdout=subprocess.PIPE, universal_newlines=True)
assert np.isclose(proc.returncode, 0)

## check the CRS and affine are written correctly
epsg = 4326
transform = (0.1, 0.0, -119.35, 0, -0.1, 35.05)
group = 'science/grids/corrections/external/troposphere'
for v in 'troposphereWet troposphereHydrostatic'.split():
with rio.open(f'netcdf:{updated_GUNW}:{group}/{v}') as ds:
assert np.isclose(ds.crs.to_epsg(), epsg), 'CRS incorrect'
assert ds.transform.almost_equals(transform), 'Affine Transform incorrect'

with xr.open_dataset(updated_GUNW, group=group) as ds:
for v in 'troposphereWet troposphereHydrostatic'.split():
da = ds[v]
assert da.rio.transform().almost_equals(transform), 'Affine Transform incorrect'

crs = rio.crs.CRS.from_wkt(ds['crs'].crs_wkt)
assert np.isclose(crs.to_epsg(), epsg), 'CRS incorrect'


# Clean up files
shutil.rmtree(SCENARIO_DIR)
os.remove('GUNW_20200130-20200124.yaml')
return
86 changes: 86 additions & 0 deletions test/test_datelist.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
import os
import glob
import pytest
import subprocess
import shutil
import yaml
import numpy as np
from test import TEST_DIR


def test_datelist():
SCENARIO_DIR = os.path.join(TEST_DIR, 'datelist')
os.makedirs(SCENARIO_DIR, exist_ok=True)
dates = ['20200124', '20200130']

dct_group = {
'aoi_group': {'bounding_box': [28, 39, -123, -112]},
'date_group': {'date_list': dates},
'time_group': {'time': '00:00:00'},
'weather_model': 'GMAO',
'runtime_group': {
'output_directory': SCENARIO_DIR,
'weather_model_directory': os.path.join(SCENARIO_DIR, 'weather_files')
}
}

params = dct_group
dst = os.path.join(SCENARIO_DIR, 'temp.yaml')

with open(dst, 'w') as fh:
yaml.dump(params, fh, default_flow_style=False)


## run raider on new file (two dates)
cmd = f'raider.py {dst}'
proc = subprocess.run(cmd.split(), stdout=subprocess.PIPE, universal_newlines=True)
assert np.isclose(proc.returncode, 0)

## check that four files (2x date) were created
n_files = len(glob.glob(os.path.join(SCENARIO_DIR, 'weather_files/*.nc')))
n_dates = len(dates)
assert np.isclose(n_files, n_dates*2), 'Incorrect number of files produced'

## clean up
shutil.rmtree(SCENARIO_DIR)

return dst


def test_datestep():
SCENARIO_DIR = os.path.join(TEST_DIR, 'datelist')
os.makedirs(SCENARIO_DIR, exist_ok=True)
st, en, step = '20200124', '20200130', 3
n_dates = 3

dct_group = {
'aoi_group': {'bounding_box': [28, 39, -123, -112]},
'date_group': {'date_start': st, 'date_end': en, 'date_step': step},
'time_group': {'time': '00:00:00'},
'weather_model': 'GMAO',
'runtime_group': {
'output_directory': SCENARIO_DIR,
'weather_model_directory': os.path.join(SCENARIO_DIR, 'weather_files')
}
}

params = dct_group
dst = os.path.join(SCENARIO_DIR, 'temp.yaml')

with open(dst, 'w') as fh:
yaml.dump(params, fh, default_flow_style=False)


## run raider on new file (two dates)
cmd = f'raider.py {dst}'
proc = subprocess.run(cmd.split(), stdout=subprocess.PIPE, universal_newlines=True)
assert np.isclose(proc.returncode, 0)

## check that four files (2x date) were created
n_files = len(glob.glob(os.path.join(SCENARIO_DIR, 'weather_files/*.nc')))
assert np.isclose(n_files, n_dates*2), 'Incorrect number of files produced'

## clean up
shutil.rmtree(SCENARIO_DIR)

return dst
2 changes: 1 addition & 1 deletion test/test_llreader.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ def test_read_station_file(station_file):
def test_bounds_from_latlon_rasters():
latfile = os.path.join(GEOM_DIR, 'lat.rdr')
lonfile = os.path.join(GEOM_DIR, 'lon.rdr')
_, snwe, _ = bounds_from_latlon_rasters(latfile, lonfile)
snwe, _, _ = bounds_from_latlon_rasters(latfile, lonfile)

bounds_true =[15.7637, 21.4936, -101.6384, -98.2418]
assert all([np.allclose(b, t, rtol=1e-4) for b, t in zip(snwe, bounds_true)])
Expand Down
12 changes: 7 additions & 5 deletions test/test_scenarios.py → test/test_scenario_1.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
import os
import pytest
import subprocess
import shutil
import glob

from test import TEST_DIR

Expand All @@ -14,15 +16,15 @@ def test_scenario_1():
process = subprocess.run(['raider.py', test_path],stdout=subprocess.PIPE, universal_newlines=True,)
assert process.returncode == 0

new_data = xr.load_dataset('HRRR_tropo_20200101T120000_ztd.nc')
golden_data = xr.load_dataset(os.path.join(SCENARIO_DIR, 'HRRR_tropo_20200101T120000_ztd.nc'))
new_data = xr.load_dataset(os.path.join(SCENARIO_DIR, 'HRRR_tropo_20200101T120000_ztd.nc'))
golden_data = xr.load_dataset(os.path.join(SCENARIO_DIR, 'golden_data', 'HRRR_tropo_20200101T120000_ztd.nc'))


assert np.allclose(golden_data['wet'], new_data['wet'])
assert np.allclose(golden_data['hydro'], new_data['hydro'])


# Clean up files
subprocess.run(['rm', '-f', './HRRR*'])
subprocess.run(['rm', '-rf', './weather_files'])

for f in glob.glob(os.path.join(SCENARIO_DIR, 'HRRR*')):
os.remove(f)
shutil.rmtree(os.path.join(SCENARIO_DIR, 'weather_files'))
11 changes: 7 additions & 4 deletions test/test_scenario_3.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
import os
import pytest
import subprocess
import shutil
import glob

from test import TEST_DIR

Expand All @@ -16,12 +18,13 @@ def test_scenario_3():
process = subprocess.run(['raider.py', test_path],stdout=subprocess.PIPE, universal_newlines=True)
assert process.returncode == 0

new_data = xr.load_dataset('HRRR_tropo_20181113T230000_ray.nc')
golden_data = xr.load_dataset(os.path.join(SCENARIO_DIR, 'HRRR_tropo_20181113T230000_ray.nc'))
new_data = xr.load_dataset(os.path.join(SCENARIO_DIR, 'HRRR_tropo_20181113T230000_ray.nc'))
golden_data = xr.load_dataset(os.path.join(SCENARIO_DIR, 'golden_data', 'HRRR_tropo_20181113T230000_ray.nc'))

assert np.allclose(golden_data['wet'], new_data['wet'])
assert np.allclose(golden_data['hydro'], new_data['hydro'])

# Clean up files
subprocess.run(['rm', '-f', './HRRR*'])
subprocess.run(['rm', '-rf', './weather_files'])
for f in glob.glob(os.path.join(SCENARIO_DIR, 'HRRR*')):
os.remove(f)
shutil.rmtree(os.path.join(SCENARIO_DIR, 'weather_files'))
Loading

0 comments on commit 2c450e0

Please sign in to comment.