resolve conflicts

CHANGELOG.md

@@ -7,6 +7,7 @@ and this project adheres to [PEP 440](https://www.python.org/dev/peps/pep-0440/)
 and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## Latest updates:
++ Use native model levels in HRRR which extend up to 2 hPa as opposed to 50 hPa in pressure levels
 + Update tests to account for different interpolation scheme
 + Dont error out when the weather model contains nan values (HRRR)
 + Fix bug in fillna3D for NaNs at elevations higher than present in the weather model
@@ -31,7 +32,7 @@ and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 + Re-work the HRRR weather model to use herbie (https://github.com/blaylockbk/Herbie) for weather model access. HRRR conus and Alaska validation periods are respectively 2016-7-15 and 2018-7-13 onwards.
 + minor bug fixes and unit test updates
 + add log file write location as a top-level command-line option and within Python as a user-specified option
-+ account for grid spacing impact on bounding box before downloading weather model 
++ account for grid spacing impact on bounding box before downloading weather model
 + update the GUNW test to account for change in grid spacing on affine transform
 + add CLI for the old processDelayFiles script and rename to raiderCombine
 + Fix gridding bug in accessing HRRR-AK 
@@ -52,15 +53,15 @@ and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 + ensure directories for storage are written
 + fix bug in writing delays for station files
 + Force lat/lon/hgt to float32 so that they line up correctly in stitching
-+ Add two stage buffer; 
++ Add two stage buffer;
     + first pad user bounding box such that a 3D cube is generated that at min covers user area of interest.
     + then if ray tracing is used, pad the downloaded model in look direction. Assumes look angle is fixed increases with latitude.
-       
+
 + Update and convert user given AOI to weather model projection (except for HRRR)
 + Clean up error messagse, skip date if temporal interpolation fails
 + Update valid range for ERA5 (current date - 3 months) & ERA5T
 + Temporal interpolation of delays if the requested datetime is more than _THRESHOLD_SECONDS away from the closest weather model available time and `interpolate_time = True` (default behavior)
-+ Add assert statement to raise error if the delay cube for each SAR date in a GUNW IFG is not written 
++ Add assert statement to raise error if the delay cube for each SAR date in a GUNW IFG is not written
 + Verify some constants / equations and remove the comments questioning them
 + Relocate the time resolution of wmodels to one spot
 + Skip test_scenario_3 until a new golden dataset is created
@@ -70,15 +71,16 @@ and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 + For the GUNW workflow:
    - Updated GUNW workflow to expose input arguments (usually passed through command line options) within the python function for testing
    - Include integration test of HRRR for GUNW workflow
-   - Test the json write (do not test s3 upload/download)
+   - Test the json write (do not test s3 upload/download) in that it conforms to the DAAC ingest schema correctly - we add a weather model field to the metadata in this workflow
    - Removed comments in GUNW test suite that were left during previous development
+   - If a bucket is provided and the GUNWs reference or secondary scenes are not in the valid range, we do nothing - this is to ensure that GUNWs can still be delivered to the DAAC without painful operator (i.e. person submitting to the hyp3 API) book-keeping
 
 ## [0.4.2]
 
 ### New/Updated Features
 + `prepFromGUNW` reads the date/time from the SLCs rather than the GUNW filename
 + `calcDelaysGUNW` allows processing with any supported weather model as listed in [`RAiDER.models.allowed.ALLOWED_MODELS`](https://github.com/dbekaert/RAiDER/blob/dev/tools/RAiDER/models/allowed.py).
-+ Removed NCMR removed from supported model list till re-tested 
++ Removed NCMR removed from supported model list till re-tested
 + `credentials` looks for weather model API credentials RC_file hidden file, and creates it if it does not exists
 + Isolate ISCE3 imports to only those functions that need it.
 + Small bugfixes and updates to docstrings
@@ -102,34 +104,34 @@ and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [0.4.0]
 
-Adding of new GUNW support to RAiDER. This is an interface delivery allowing for subsequent integration into HYP3 (input/output parsing is not expected to change; computed data is not yet verified). 
+Adding of new GUNW support to RAiDER. This is an interface delivery allowing for subsequent integration into HYP3 (input/output parsing is not expected to change; computed data is not yet verified).
 
 ### New/Updated Features
 + Working GUNW entry point in workflow for raider.py
 + Ability to parse a GUNW to workflows from which all required RAiDER information is extracted (e.g. dates, UTC, orbit, bbox, look direction, wavelength) with an option to specify weather model (those already supported by RAiDER) and ability to squeeze in the derived output into the original GUNW product.
-+ Delays for GUNW are calculated in RAiDER using the ray-tracing option specifying bbox (GUNW driven), a hardcoded lateral posting (0.05º for HRRR and 0.1º for others),  fixed vertical height levels, using an different orbit file for  secondary and master. 
++ Delays for GUNW are calculated in RAiDER using the ray-tracing option specifying bbox (GUNW driven), a hardcoded lateral posting (0.05º for HRRR and 0.1º for others),  fixed vertical height levels, using an different orbit file for  secondary and master.
      - The hard-coded heights and posting will be refined per model and to ensure stitching abilities in ARIA-tools.
-     - The orbit should be refined to not change between secondary and reference to avoid issues. See https://github.com/dbekaert/RAiDER/discussions/435#discussioncomment-4392665 
-+ Bug fix for raider.py "date" input argument when multiple dates are requested (i.e. support of requesting two dates or two dates with a sampling).  
+     - The orbit should be refined to not change between secondary and reference to avoid issues. See https://github.com/dbekaert/RAiDER/discussions/435#discussioncomment-4392665
++ Bug fix for raider.py "date" input argument when multiple dates are requested (i.e. support of requesting two dates or two dates with a sampling).
 + Add unit test for date input argument checking (single day, two dates, two dates with samples)
 + Write the diagnostic weather model files to the 'output_directory' rather than PWD
 + Fix for incorrectly written hard-cored projection embedded in the computed output data
 + Allow for multiple orbits files/dates to be used for slant:projection
-+ correctly pass llh to lla_to_ecef function for slant:projection 
++ correctly pass llh to lla_to_ecef function for slant:projection
     ++ verified this doesnt change anything
 + removed deprecated ray projection functionality
 + added 1º buffer for zenith and projected (already done for ray tracing)
 + differential delay is rounded to model-dependent nearest hour
-+ version 1c hardcoded into the updated GUNW 
++ version 1c hardcoded into the updated GUNW
 
 ### Added dependencies for:
-+ sentinelof: used to fetch the orbit for GUNW 
++ sentinelof: used to fetch the orbit for GUNW
 + rioxarray: used for reading rasters with xarray
 
 ### Not implemented / supported in this release### Not implemented / supported in this release
-+ no temporal interpolation 
++ no temporal interpolation
 + no refined model specific hardcoded spacing and heights
-+ no ability for single orbit Interferometric calculation 
++ no ability for single orbit Interferometric calculation
 + no verification of results
 
 ## [0.3.1]

setup.py

@@ -44,4 +44,5 @@
 setup(
     ext_modules=cython_extensions + pybind_extensions,
     cmdclass={"build_ext": build_ext},
+    package_data={'tools': ['RAiDER/models/*.zip']}
 )

test/conftest.py

@@ -1,4 +1,5 @@
 from pathlib import Path
+from typing import Callable
 
 import pytest
 
@@ -30,8 +31,16 @@ def test_dir_path() -> Path:
 
 
 @pytest.fixture(scope='session')
-def test_gunw_path() -> Path:
-    return TEST_DIR / 'gunw_test_data' / 'S1-GUNW-D-R-071-tops-20200130_20200124-135156-34956N_32979N-PP-913f-v2_0_4.nc'
+def test_gunw_path_factory() -> Callable:
+    def factory(location: str = 'california-t71') -> Path:
+        if location == 'california-t71':
+            file_name = 'S1-GUNW-D-R-071-tops-20200130_20200124-135156-34956N_32979N-PP-913f-v2_0_4.nc'
+        elif location == 'alaska':
+            file_name = 'S1-GUNW-D-R-059-tops-20230320_20220418-180300-00179W_00051N-PP-c92e-v2_0_6.nc'
+        else:
+            raise NotImplementedError
+        return TEST_DIR / 'gunw_test_data' / file_name
+    return factory
 
 
 @pytest.fixture(scope='session')

test/test_GUNW.py

@@ -6,14 +6,19 @@
 import unittest
 
 import jsonschema
+<<<<<<< HEAD
 import numpy as np
+=======
+import pandas as pd
+>>>>>>> dev
 import pytest
 import rasterio as rio
 import xarray as xr
 
 import RAiDER
 import RAiDER.cli.raider as raider
 from RAiDER import aws
+from RAiDER.aria.prepFromGUNW import check_weather_model_availability
 from RAiDER.cli.raider import calcDelaysGUNW
 
 
@@ -29,11 +34,11 @@ def compute_transform(lats, lons):
 
 
 @pytest.mark.isce3
-@pytest.mark.parametrize('weather_model_name', ['GMAO', 'HRRR'])
-def test_GUNW_update(test_dir_path, test_gunw_path, weather_model_name):
+@pytest.mark.parametrize('weather_model_name', ['GMAO'])
+def test_GUNW_update(test_dir_path, test_gunw_path_factory, weather_model_name):
     scenario_dir = test_dir_path / 'GUNW'
     scenario_dir.mkdir(exist_ok=True, parents=True)
-    orig_GUNW = test_gunw_path
+    orig_GUNW = test_gunw_path_factory()
     updated_GUNW = scenario_dir / orig_GUNW.name
     shutil.copy(orig_GUNW, updated_GUNW)
 
@@ -80,6 +85,7 @@ def test_GUNW_metadata_update(test_gunw_json_path, test_gunw_json_schema_path, t
     mocker.patch("RAiDER.aws.get_s3_file", side_effect=['foo.nc', temp_json_path])
     mocker.patch("RAiDER.aws.upload_file_to_s3")
     mocker.patch("RAiDER.aria.prepFromGUNW.main", return_value=['my_path_cfg', 'my_wavelength'])
+    mocker.patch("RAiDER.aria.prepFromGUNW.check_weather_model_availability", return_value=True)
     mocker.patch("RAiDER.cli.raider.calcDelays", return_value=['file1', 'file2'])
     mocker.patch("RAiDER.aria.calcGUNW.tropo_gunw_slc")
     mocker.patch("os.getcwd", return_value='myDir')
@@ -92,7 +98,7 @@ def test_GUNW_metadata_update(test_gunw_json_path, test_gunw_json_schema_path, t
     metadata = json.loads(temp_json_path.read_text())
     schema = json.loads(test_gunw_json_schema_path.read_text())
 
-    assert metadata['weather_model'] == ['HRES']
+    assert metadata['metadata']['weather_model'] == ['HRES']
     assert (jsonschema.validate(instance=metadata, schema=schema) is None)
 
     assert aws.get_s3_file.mock_calls == [
@@ -148,3 +154,64 @@ def test_azimuth_timing_against_interpolation(model, tmp_path, gunw_azimuth_test
             diff_mm = (da_1 - da_0).data * 0.055465761572122574 / (4 * np.pi) * 1_000
             # Differences in mm are bounded by 1
             assert np.all(diff_mm < 1)
+@pytest.mark.parametrize('weather_model_name', ['GMAO', 'HRRR', 'HRES', 'ERA5', 'ERA5'])
+def test_check_weather_model_availability(test_gunw_path_factory, weather_model_name, mocker):
+    # Should be True for all weather models
+    # S1-GUNW-D-R-071-tops-20200130_20200124-135156-34956N_32979N-PP-913f-v2_0_4.nc
+    test_gunw_path = test_gunw_path_factory()
+    assert check_weather_model_availability(test_gunw_path, weather_model_name)
+
+    # Let's mock an earlier date for some models
+    mocker.patch("RAiDER.aria.prepFromGUNW.get_acq_from_slc_id", side_effect=[pd.Timestamp('2015-01-01'),
+                                                                              pd.Timestamp('2014-01-01')])
+    cond = check_weather_model_availability(test_gunw_path, weather_model_name)
+    if weather_model_name in ['HRRR', 'GMAO']:
+        cond = not cond
+    assert cond
+
+
+@pytest.mark.parametrize('weather_model_name', ['GMAO', 'HRRR'])
+def test_check_weather_model_availability_over_alaska(test_gunw_path_factory, weather_model_name, mocker):
+    # Should be True for all weather models
+    # S1-GUNW-D-R-059-tops-20230320_20220418-180300-00179W_00051N-PP-c92e-v2_0_6.nc
+    test_gunw_path = test_gunw_path_factory(location='alaska')
+    assert check_weather_model_availability(test_gunw_path, weather_model_name)
+
+    # Let's mock an earlier date
+    mocker.patch("RAiDER.aria.prepFromGUNW.get_acq_from_slc_id", side_effect=[pd.Timestamp('2017-01-01'),
+                                                                              pd.Timestamp('2016-01-01')])
+    cond = check_weather_model_availability(test_gunw_path, weather_model_name)
+    if weather_model_name == 'HRRR':
+        cond = not cond
+    assert cond
+
+
+@pytest.mark.parametrize('weather_model_name', ['HRRR', 'GMAO'])
+@pytest.mark.parametrize('location', ['california-t71', 'alaska'])
+def test_weather_model_availability_integration(location, test_gunw_path_factory, tmp_path, weather_model_name, mocker):
+    temp_json_path = tmp_path / 'temp.json'
+    test_gunw_path = test_gunw_path_factory(location=location)
+    shutil.copy(test_gunw_path, temp_json_path)
+
+    # We will pass the test GUNW to the workflow
+    mocker.patch("RAiDER.aws.get_s3_file", side_effect=[test_gunw_path, 'foo.json'])
+    mocker.patch("RAiDER.aws.upload_file_to_s3")
+    # These are outside temporal availability of GMAO and HRRR
+    ref_date, sec_date = pd.Timestamp('2015-01-01'), pd.Timestamp('2014-01-01')
+    mocker.patch("RAiDER.aria.prepFromGUNW.get_acq_from_slc_id", side_effect=[ref_date, sec_date])
+    # Don't specify side-effects or return values, because never called
+    mocker.patch("RAiDER.aria.prepFromGUNW.main")
+    mocker.patch("RAiDER.cli.raider.calcDelays")
+    mocker.patch("RAiDER.aria.calcGUNW.tropo_gunw_slc")
+
+    iargs = ['--weather-model', weather_model_name,
+             '--bucket', 'myBucket',
+             '--bucket-prefix', 'myPrefix']
+    out = calcDelaysGUNW(iargs)
+    # Check it returned None
+    assert out is None
+
+    # Check these functions were not called
+    RAiDER.cli.raider.calcDelays.assert_not_called()
+    RAiDER.aria.prepFromGUNW.main.assert_not_called()
+    RAiDER.aria.calcGUNW.tropo_gunw_slc.assert_not_called()

test/test_HRRR_ztd.py

@@ -18,7 +18,7 @@ def test_scenario_1():
 
     new_data  = xr.load_dataset(os.path.join(SCENARIO_DIR, 'HRRR_tropo_20200101T120000_ztd.nc'))
     new_data1 = new_data.sel(x=-91.84, y=36.84, z=0, method='nearest')
-    golden_data = 2.2011017, 0.03755665 # hydro|wet
+    golden_data = 2.2622863, 0.0361021 # hydro|wet
 
     np.testing.assert_almost_equal(golden_data[0], new_data1['hydro'].data)
     np.testing.assert_almost_equal(golden_data[1], new_data1['wet'].data)

tools/RAiDER/aria/prepFromGUNW.py

@@ -10,20 +10,102 @@
 import numpy as np
 import xarray as xr
 import rasterio
+import geopandas as gpd
+import pandas as pd
 import yaml
 import shapely.wkt
 from dataclasses import dataclass
+import sys
+from shapely.geometry import box
+from rasterio.crs import CRS
 
 import RAiDER
 from RAiDER.utilFcns import rio_open, writeArrayToRaster
 from RAiDER.logger import logger
 from RAiDER.models import credentials
+from RAiDER.models.hrrr import HRRR_CONUS_COVERAGE_POLYGON, AK_GEO
 from eof.download import download_eofs
 
 ## cube spacing in degrees for each model
 DCT_POSTING = {'HRRR': 0.05, 'HRES': 0.10, 'GMAO': 0.10, 'ERA5': 0.10, 'ERA5T': 0.10}
 
 
+def get_slc_ids_from_gunw(gunw_path: str,
+                          reference_or_secondary: str = 'reference') -> list[str]:
+    if reference_or_secondary not in ['reference', 'secondary']:
+        raise ValueError('"reference_or_secondary" must be either "reference" or "secondary"')
+    group = f'science/radarMetaData/inputSLC/{reference_or_secondary}'
+    with xr.open_dataset(gunw_path, group=group) as ds:
+        slc_ids = ds['L1InputGranules'].values
+    return slc_ids
+
+
+def get_acq_from_slc_id(slc_id: str) -> pd.Timestamp:
+    ts_str = slc_id.split('_')[5]
+    return pd.Timestamp(ts_str)
+
+
+def check_weather_model_availability(gunw_path: str,
+                                     weather_model_name: str) -> bool:
+    """Checks weather reference and secondary dates of GUNW occur within
+    weather model valid range
+
+    Parameters
+    ----------
+    gunw_path : str
+    weather_model_name : str
+        Should be one of 'HRRR', 'HRES', 'ERA5', 'ERA5T', 'GMAO'.
+    Returns
+    -------
+    bool:
+        True if both reference and secondary acquisitions are within the valid range. We assume that
+        reference_date > secondary_date (i.e. reference scenes are most recent)
+
+    Raises
+    ------
+    ValueError
+        - If weather model is not correctly referencing the Class from RAiDER.models
+        - HRRR was requested and it's not in the HRRR CONUS or HRRR AK coverage area
+    """
+    ref_slc_ids = get_slc_ids_from_gunw(gunw_path, reference_or_secondary='reference')
+    sec_slc_ids = get_slc_ids_from_gunw(gunw_path, reference_or_secondary='secondary')
+
+    ref_ts = get_acq_from_slc_id(ref_slc_ids[0])
+    sec_ts = get_acq_from_slc_id(sec_slc_ids[0])
+
+    if weather_model_name == 'HRRR':
+        group = '/science/grids/data/'
+        variable = 'coherence'
+        with rasterio.open(f'netcdf:{gunw_path}:{group}/{variable}') as ds:
+            gunw_poly = box(*ds.bounds)
+        if HRRR_CONUS_COVERAGE_POLYGON.intersects(gunw_poly):
+            pass
+        elif AK_GEO.intersects(gunw_poly):
+            weather_model_name = 'HRRRAK'
+        else:
+            raise ValueError('HRRR was requested but it is not available in this area')
+
+    # source: https://stackoverflow.com/a/7668273
+    # Allows us to get weather models as strings
+    # getattr(module, 'HRRR') will return HRRR class
+    module = sys.modules['RAiDER.models']
+    weather_model_names = module.__all__
+    if weather_model_name not in weather_model_names:
+        raise ValueError(f'The "weather_model_name" must be in {", ".join(weather_model_names)}')
+
+    weather_model_cls = getattr(module, weather_model_name)
+    weather_model = weather_model_cls()
+
+    wm_start_date, wm_end_date = weather_model._valid_range
+    if isinstance(wm_end_date, str) and wm_end_date == 'Present':
+        wm_end_date = datetime.today() - weather_model._lag_time
+    elif not isinstance(wm_end_date, datetime):
+        raise ValueError(f'the weather model\'s end date is not valid: {wm_end_date}')
+    ref_cond = ref_ts <= wm_end_date
+    sec_cond = sec_ts >= wm_start_date
+    return ref_cond and sec_cond
+
+
 @dataclass
 class GUNW:
     path_gunw: str