SDAP-521 - Improved SDAP testing suite #325
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 24 commits
July 1, 2024 09:24
Made current with what was deployed for the CDMS project. Will need extensive editing.
This reverts commit 46cf5ad.
# Conflicts: # docs/index.rst # docs/release.rst
ngachung
approved these changes
Oct 16, 2024
RKuttruff
added a commit
that referenced
this pull request
Nov 4, 2024
* SDAP-493: Pagination improvements (#282) * removed resultSizeLimit param from matchup * Add # of primaries/avergae secondaries to job output * rename to executionId * update changelog * add totalSecondaryMatched field to /job output * num unique secondaries addition * updated docs to use correct sea_water_temperature param name * bugfix * fix division by zero bug * SDAP-500: Improvements to SDAP Asynchronous Jobs (#287) * removed resultSizeLimit param from matchup * Add # of primaries/avergae secondaries to job output * rename to executionId * update changelog * add totalSecondaryMatched field to /job output * num unique secondaries addition * updated docs to use correct sea_water_temperature param name * bugfix * fix division by zero bug * pagination improvements * removed debugging line * changelog * Update helm cassandra dependency (#289) * Update helm cassandra dependency * Bump default cassandra PV to 4 * Bump default cassandra PV to 4 in tools * Changelog * Fixed small documentation issue --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * update changelog * rename key to executionID --------- Co-authored-by: Riley Kuttruff <72955101+RKuttruff@users.noreply.github.com> Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * SDAP-499: Add page number to default filename for matchup output (#288) * removed resultSizeLimit param from matchup * Add # of primaries/avergae secondaries to job output * rename to executionId * update changelog * add totalSecondaryMatched field to /job output * num unique secondaries addition * updated docs to use correct sea_water_temperature param name * bugfix * fix division by zero bug * add page number to default filename for matchup output * pagination improvements * removed debugging line * changelog * Update helm cassandra dependency (#289) * Update helm cassandra dependency * Bump default cassandra PV to 4 * Bump default cassandra PV to 4 in tools * Changelog * Fixed small documentation issue --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * Revert "Update helm cassandra dependency (#289)" This reverts commit 1e8cc4e. * changelog --------- Co-authored-by: Riley Kuttruff <72955101+RKuttruff@users.noreply.github.com> Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * SDAP-506: STAC Catalog for Matchup outputs (#291) * removed resultSizeLimit param from matchup * Add # of primaries/avergae secondaries to job output * rename to executionId * update changelog * add totalSecondaryMatched field to /job output * num unique secondaries addition * updated docs to use correct sea_water_temperature param name * bugfix * fix division by zero bug * add page number to default filename for matchup output * pagination improvements * removed debugging line * changelog * Update helm cassandra dependency (#289) * Update helm cassandra dependency * Bump default cassandra PV to 4 * Bump default cassandra PV to 4 in tools * Changelog * Fixed small documentation issue --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * stac catalog * Updated openapi spec * move stac endpoints to matchup tag in openapi spec * Revert "Update helm cassandra dependency (#289)" This reverts commit 1e8cc4e. * fix bug where still-running jobs failed /job endpoint due to missing metadata * Update .asf.yaml (#293) Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * update changelog * re-add removed changelog entry --------- Co-authored-by: Riley Kuttruff <72955101+RKuttruff@users.noreply.github.com> Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * SDAP-508 report sat spatial extents (#295) Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * SDAP-505: Support DOMS insitu API (#299) * support DOMS insitu API * update NCAR insitu url * add endpoint for getting insitu units * SDAP-472 - data-access overhaul to support multiple simultaneous data backends (#294) * Separated NTS backends * n/a * More nts backend stuff * Working(?) np backend * Working(?) np backend * gitignore ini * ASF headers * First functioning test of 2 simultaneous backends * Removed accidentally committed ini files * Working zarr backend ds list + datasets are no longer case sensitive + handling for failed zarr ds opens (bad path, bad creds, &c) * Capture and handle NTS requests routed to backend that doesn't (yet) support them * analysis setup fails to find VERSION.txt when building locally * Implemented more NTS functions in zarr backend * Added misc backend time metrics record field in NCSH * fixes * Dynamic dataset management * Dynamic dataset management * Dataset management * Timeseriesspark support * Update backend dict on dataset mgmt query * Fixes and improvements * Adapted matchup to work with zarr backends * Zarr support - Distinct slices of time is now default - No longer assuming+shaping as multivar tiles unless needed * DDAS adjustments * find_tile_by_polygon_and_most_recent_day_of_year impl * Don't sel by time if neither max nor min time are given * Fix not calling partial when needed * Pinned s3fs and fsspec versions * Fixed some dependencies to ensure image builds properly + s3fs works * Config override for backends * Deps update * Add metadata from Zarr collection to /list * Zarr: Probe lat order and flip if necessary * Strip quotes from variable names CM can sometimes publish with extra quotes resulting in KeyErrors * removed resultSizeLimit param from matchup * Add # of primaries/avergae secondaries to job output * rename to executionId * update changelog * add totalSecondaryMatched field to /job output * num unique secondaries addition * updated docs to use correct sea_water_temperature param name * bugfix * fix division by zero bug * add params to dataset management handler classes * add page number to default filename for matchup output * pagination improvements * removed debugging line * changelog * Update helm cassandra dependency (#289) * Update helm cassandra dependency * Bump default cassandra PV to 4 * Bump default cassandra PV to 4 in tools * Changelog * Fixed small documentation issue --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * stac catalog * Updated openapi spec * move stac endpoints to matchup tag in openapi spec * SDAP-507 - Changes to remove geos sub-dependency * SDAP-507 - Changelog * SDAP-507 - Changes to remove geos sub-dependency * SDAP-507 - Changelog * delete instead of comment out * Revert "Update helm cassandra dependency (#289)" This reverts commit 1e8cc4e. * deleted disabled endpoint files * fix bug where still-running jobs failed /job endpoint due to missing metadata * Update .asf.yaml (#293) Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * Moved changelog entries * SDAP-472 changelog entries --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> Co-authored-by: skorper <stepheny.k.perez@gmail.com> * Merge master into develop post-1.2.0 release (#305) * Update .asf.yaml (#293) Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * domspurge patch (#280) * Fix delete query to account for PK update * Fixed mistake in domspurge readme --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * Patch: Made conda dep changes accidentally omitted from geos PR * SDAP-511 - Switch package manager to poetry (#301) * Redid branch w/ correct base I tried to just rebase but it didn't work properly * Changelogs * Delete old dockerfile * Removed old conda install files * Removed version.txt to just use pyproject.toml --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * pyproject.toml dependency updates for Zarr --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> Co-authored-by: Stepheny Perez <skorper@users.noreply.github.com> * SDAP 515 - Improved handling of unreachable remote SDAPs (#308) * Improved error handling to account for any reason a remote is not reachable * Further fixes for unreachable remote SDAPs * Improved logging * SDAP-513 - Configurable Solr init pod image in helm (#306) Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * Added some missing ASF headers and removed some unneeded files (#302) * Remove IDE files * Added more missing headers --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * Update requirements.txt (#309) * Updates to openapi (#310) * Disable the try it now button * Added additional spark algorithms * Updates to openapi to include additional algorithms * SDAP-518 - Collection Config Docs (#311) * Initial work on CC docs * Add collections to index toctree * YAML highlighting * NetCDF section * remove incubation msg from intro.rst * Added recs for gridded tile size --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * data-access patch (#313) * Patches to backend mgmt and zarr backend - Code cleanup - Added dask as dependency, so it will be leveraged with Zarr datasets - Fixed creation of tile times array from Zarr data to ensure its in seconds from 1970-01-01. Original method has been seen to produce incorrect results from bad assumptions * Changelog --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * Changed SDAP startup behavior to wait for all datasets to be prepared before accepting HTTP requests (#314) Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * update quickstart with updated solr command (#298) Co-authored-by: Riley Kuttruff <72955101+RKuttruff@users.noreply.github.com> * 1.3.0 to develop (#317) * Update .asf.yaml (#293) Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * domspurge patch (#280) * Fix delete query to account for PK update * Fixed mistake in domspurge readme --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * Patch: Made conda dep changes accidentally omitted from geos PR * SDAP-511 - Switch package manager to poetry (#301) * Redid branch w/ correct base I tried to just rebase but it didn't work properly * Changelogs * Delete old dockerfile * Removed old conda install files * Removed version.txt to just use pyproject.toml --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * Update release files - 1.3.0 header in changelogs - Removed incubation DISCLAIMER - Removed incubation language from NOTICE and updated URLs - Bumped versions - Made copyright year current - Poetry re-lock * Fix Dockerfile for DISCLAIMER rm --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> Co-authored-by: Stepheny Perez <skorper@users.noreply.github.com> * SDAP-469 - Support for data with elevation (#276) * SDAP-469 - Configured timeseriesspark to have the option to bound by depth * Elevation fetch & masking untested * Elevation subsetting * fixes * Check if there's elevation when pulling the tile data * Elevation masking for tiles with no elevation does nothing * Add elevation fields to collection creation script * Name elevation fields to be dynamically typed correctly * Elevation array shape + add to nexuspoint * Add elev to data in bounds result * Improved handling of elev arguments & masking * tile elev in nexusproto backend * Docker fix * Additional algorithm support for SDAP 469 3d data (#10) * Added support for default elevation value of 0 when parameter not provided * Added support for setting min and max elevation when using single elevation arg * Added support for data ingested with elevation * Fixed logic for elevation clause when min and max elevations are equivalent * Fixed logic for elevation clause when min and max elevations are equivalent * Bug fix for missing elevation parameters * Reverted timeAvgMapSpark to use NexusRequestObject * Fixed bug with order of arguments * Commented out saving netcdf files * Bug fix for how numpy arrays are handled * Cleaned up logic for handling the different arg cases * Reworked elevation clause logic and added to polygon search * Added elevation args to find_tiles_in_polygon --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> Co-authored-by: Kevin <kevinmarlis@gmail.com> * SDAP-522 - Fixes for broken endpoints found by work for SDAP-521 (#319) * Implemented fixes * Changelog --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * Error handling on ds listing for zarr (#320) Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * SDAP-319 - Fixed typo (#322) * Added forgotten changelog entry (#321) Forgot from PR #276 * [SDAP-492] STV-FIS tomogram data visualization support (#323) * SDAP-469 - Configured timeseriesspark to have the option to bound by depth * Elevation fetch & masking untested * Elevation subsetting * fixes * Check if there's elevation when pulling the tile data * Elevation masking for tiles with no elevation does nothing * SDAP-492 - Added dependencies + alg classes * Initial subsetting Still need to org and plot the data * Image rendering * Lon + Lat tomo fetch impls * Add elevation fields to collection creation script * Name elevation fields to be dynamically typed correctly * Elevation array shape + add to nexuspoint * WIP: Optimizations for lateral slicing (lon only so far) Untested * Elevation array shape + add to nexuspoint * Fixes * bugfix * Use actual extent in extent param * Minor improvements * More tomogram work * Render profile tomo image results using `plt.pcolormesh` instead of `plt.imshow` * SIGNIFICANT improvement of tomogram processing time * Add elev to data in bounds result * Add elev to data in bounds result * Added GIF renderer * 3D tomogram viz endpoint * Add labels * Colorbars * Add basemap below tomo render * Additional arguments and render types * Simple CSV renderer - point cloud only * Improved handling of elev arguments & masking * tile elev in nexusproto backend * poetry re-lock after deps update from merge * minor updates * Docker fix * Speed up 3d result build * poetry relock * Vertical truncation of tomograms by RH98/GND height maps * Updates: - Disabled elev tomo endpoint - Added param to render tomograms in vertically cumulative percentiles. (ie, cumulative return power for each voxel column) - Added NetCDF renderer for tomo3d endpoint * Tomo improvements * Fix for tomo 3d basemap being inverted by latitude * poetry lock * Simplified profile tomo API + added NC output formatter * Added multi-slice capability to 2d plotting * Add cbar min/max option to tomo profile rendering + better nodata error handling * Support for arbitrary line slicing of tomogram data Currently does not support multislicing * Remove accidentally committed test code * Add slicing along line to NTS w/ nexusproto implementation + tomogram arb slicing tries to use this with bbox subsetting as a fallback * Improve performance of elevation binning * Improve performance of data gridding * Update CL + remove LGPL 3 indirect dependency --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * [SDAP-497] - Release build script (#290) * Build script * Build script * Rebased onto 1.2.0 release branch to avoid messing up changelog * Retry failed builds/pushes once to avoid time loss from transient errors * Moved changelog entry * Start of src pull changes * mv build script * Better ASF pulling * Build from GitHub * Simple docs * Some cleanup, renaming, deleting unused stuff etc * Update build.py * Added build from ASF archive * Updates and fixes * Updates for post-grad changes * Add support for custom nexusproto builds from git * Build tool updates * Update changelog * Implement build from local directory --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * SDAP-520 - Guide for evaluating SDAP release candidates (#318) * SDAP-520 Added RC eval guide to RTD * remove incubator * add to toctree * Attempt to fix the many Sphinx warnings on build * Changelog --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * added primitive match return function to CDMS reader (#260) * added primitive match return function * Updated function to be used with generic use cases * Updated documentation for assemble functions * Updated documentation, removed xarray dependency * merged with develop, updated changelog * SDAP-470 Fixed CHANGELOG.md (#324) * SDAP-526 - Upgrade canopy and ground masking (#326) * Upgrade canopy and ground masking * Changelog update --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * Patch to add geopy dep that was forgotten (#328) Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * SDAP-527: Fixed creation of execution status Cassandra table (#329) Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * SDAP-521 - Improved SDAP testing suite (#325) * SDAP-520 Added RC eval guide to RTD * remove incubator * add to toctree * Granule download script for testing * Move & update test script Made current with what was deployed for the CDMS project. Will need extensive editing. * Test script pruning + guide start * Updates * Updates * Guide for install and run * Attempt to fix the many Sphinx warnings on build * Fix bad ref * Fix bad ref * Fix bad ref (third time's the charm?) * Removal of ingested test data * Reduced datainbounds L2 test bbox to ease memory footprint * Revert "Reduced datainbounds L2 test bbox to ease memory footprint" This reverts commit 46cf5ad. * Update docs for missing test collection * SDAP-521 Updated quickstart and test guide. (#327) * SDAP-521 Updated quickstart and test guide. * SDAP-521 Updated solr start up env variables to be consistent with helm chart. * SDAP-521 Updated README.md --------- Co-authored-by: rileykk <rileykk@jpl.nasa.gov> Co-authored-by: Nga Chung <17833879+ngachung@users.noreply.github.com> * SDAP-529: Added configuration for verbose logging for collection manager in the Helm chart (#330) Co-authored-by: rileykk <rileykk@jpl.nasa.gov> * Bump versions for release * Some more version fixes * Some more version fixes * Fix doc typo * I'm having issues with the venv. This needs more testing but shouldn't be a release blocker * Final release dates to changelog * Moved incorrect CL entry to proper section --------- Co-authored-by: Stepheny Perez <skorper@users.noreply.github.com> Co-authored-by: rileykk <rileykk@jpl.nasa.gov> Co-authored-by: skorper <stepheny.k.perez@gmail.com> Co-authored-by: Kevin <kevinmarlis@gmail.com> Co-authored-by: alovett-COAPS <113188918+alovett-COAPS@users.noreply.github.com> Co-authored-by: Nga Chung <17833879+ngachung@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Guide preview