Ideas to implement for v0.2

[mgdenno]

Going into mid-May 2023 we have built some significant functionality to support exploratory evaluation. We have focused on loading, caching, querying and visualization, and have the basics working. We hit milestone 1 for the CIROH Workshop in May 2023. Now we need to decide what to focus on next. Lets collect ideas. Please keep them one idea to a comment so we can vote on them and track "up votes". @kvanwerkhoven @samlamont @gpark69

[mgdenno]

Enhance tools to work more seamlessly with cloud storage. This would include updating the DuckDB queries to query Parquet files in S3/GCP. Could also include work to reduce the amount of data that we cache by operating on directly in bucket.

[mgdenno]

I think this might be best served by creating a class that sets up a "query" and has methods to validate, pre-process timestep dependent attributes, query, etc. This could look like:

qry = Query(
    primary_filepath: str,
    secondary_filepath: str,
    crosswalk_filepath: str,
    geometry_filepath: Union[str, None] = None,
    aws_access_key: str,
    aws_secret_key: str,
    aws_region: str
)
qry.validate()
qry.pre_process()
metrics = qry.get_metrics(
    group_by: List[str],
    order_by: List[str],
    include_metrics: Union[List[tmq.MetricEnum], "all"],
    filters: Union[List[dict], None] = None,
    return_query: bool = False,
    include_geometry: bool = False,
)

[mgdenno]

Add additional queries and metrics to support a broader range of needs. This probably means a significant refactor of our current query modules(s). What ones are needed most?

[mgdenno]

Look into and implement better utilization of Parquet files. It appears that "wider tables" might have benefit over our current "tall table" format. This may have impacts on performance, but is not a small amount of work.

[mgdenno]

This could include generating a "joined" table that includes calculated fields, that would be one less step that needs to happen "on the fly" which would improve performance too.

[mgdenno]

Build out reusable visualization components. What reusable components would be needed most?

[mgdenno]

Integrate with NextGen (particularly NextGen-in-a-Box). We could enhance the TEEHR to enable launching a container next to the NextGen output to explore from the S3 bucket where the output is. This could be deployed via terraform by NextGen users. This would be powerful.

[mgdenno]

This probably requires tight coordination with James and Co.

[mgdenno]

Implement data validation for cache data. Currently there are no check on the data and it is possible to have duplicate data without knowing it. The Parquet files to not have unique constraints like a relational database would.

[mgdenno]

Prepare and stage commonly needed datasets, for example this could include:

• USGS gages and associated hydrologic attributes,
• HUC2-HUC12 polygons and weights files
• Baseline hindcasts and forecasts to compare new model formulations to

[mgdenno]

Add attributes to query output. Make queries more robust with respect to including additional data. This includes location attributes (e.g., drainage area) and timestep calculations (e.g., was a threshold exceeded). This might be two separate features...probably is.

[mgdenno]

Build out a web API to support "normal" web applications. If this were done, some example web applications (dashboards) would also be needed.

[mgdenno]

Investigate tighter integration with NetCDF files. Not sure how this can really be done, but given how prominent these files are, might be worth a second look.

[mgdenno]

Setup deployment code. Infrastructure-as-code, deployable container, etc. such that we can deploy our own "2i2c" type service.