Versioning of the predefined catalogs

[savente93]

To determine which catalogs to read, hydromt currently accesses the predefined_catalogs.yml file from the main branch on GitHub. This file includes catalogs like the Deltares data catalog. This means that the predefined_catalogs.yml is a single point failure for the entire system. We have already experienced an outage because of this recently which concerns me. Not to mention that if we ever need to update the schema of predefined_catalogs.yml for any reason this will mean a breaking change for literally all active users. Given that we are making breaking changes in v1, let us consider this an opportunity to adjust this as necessary as well to allow ourselves more flexibility in the future. Below, I will list possibilities with some pros and cons:

Packaging the data catalogs with releases

This basically just involves including the predefined catalog.yml and any relevant catalogs as artifacts in the python package.

Pros:

1. users always have a compatible version of the data catalogs with their installation
2. Since user has the catalogs locally this cut's out the need for several round trips to GitHub improving latency significantly
3. Developers have more safety during development and have less risk of accidentally breaking the workflows of existing users

Cons:

1. Users might have to upgrade their code to use a new data catalog version. Note here that there is always the escape hatch of just downloading the new catalog version and using that directly, but this is more hassle.
2. It is difficult to add patches to data catalogs for previous versions
3. Should we ever want to start releasing executables of HydroMT with something like pyinstaller, it becomes effectively impossible

Packaging the data catalogs as their own python packages and specifying them as dependencies

Pros:

1. we get version specification for free, since the dependencies will specify compatible versions and pip/conda/whatever will do the rest
2. we are free to update data catalogs independently of the code.
3. we can release patches, yank versions, and major versions more incrementally
4. once the correct version is installed it will again save round trips to GitHub which will improve latency

Cons:

1. We give up some control over which versions users are using
2. This might be a blocker if we ever want to distribute executables, since then users will not have access to things like pip anymore outside of the executable.

Adding release branches to the HydroMT git repo.

This is similar to the current approach except that we would have dedicated release branches that the catalogs would be pulled from. think hydromt/data-version-x.y this could be done in conjunction for having release branches for the code as well so we could still release patches for previous versions, but they do not necessarily have to go together.

Pros

1. we can still patch/update/release any version of the catalog and not break everyone's workflow at the same time.
2. similar approach to the one currently in use so minimal code changes would be necessary
3. Again provides a better layer of isolation between developers and users, so we are not at as much risk of accidentally breaking their workflows

Cons

1. If we find an issue that affects multiple versions, we will have to update test and release each of them individually
2. We still have to build and maintain all the code necessary to pick compatible versions ourselves.

Building a service exposed through a REST api that resolves these issues for the user

Pros

1. This gives us maximum flexibility to update, release, maintain, support etc. all possible versions of the catalog at the same time, since we can do basically anything we like on the backend.

Cons

1. we now have to ensure availability, maintian, secure and pay for this new service
2. will have to be built from scratch

@deltamarnix did I miss anything from our discussion earlier?

[DirkEilander]

I missed this RFC earlier, thanks for writing this up! I agree with most pros and cons, it would be nice to also add the pros and cons of the current setup to make a decision if and how we would like to see this changed. I've tried to add these below and also an additional hybrid option to include the predifined_catalogs, but still host the actual catalogs online. Also we can (and should) make sure that the data catalog also works if no predefined catalogs are found.

Current setup

Pros

1. Flexibility for users to select the data catalog version independent of the hydromt version (although bounded by independent compatibility)
2. Continuous release of data catalogs independent of hydromt releases

Cons

1. Single point failure for the entire system.
2. impossible to change the schema of predefined_catalogs.yml without breaking changes

Package only predefined catalogs

The actual catalog files will still be hosted in git.

Pros

1. Flexibility for users to select order data catalog version
2. The predefined catalog should only include compatible version, so no need to check compatibility on runtime
3. We can change the scheme if needed and it would be easier for plugins to also extend the predefined catalogs.

Cons

1. For newer catalog versions we need to release hydromt with updated predefined catalogs.
2. Still dependent on availability of git (or wherever the data catalog themselves are hosted).

[deltamarnix]

Besides the discussion on where we want to host the file, and how we want to keep compatibility working, we can also consider when the catalog must be updated. For example, by making a CLI command to hydromt update_catalog, instead of just doing it on every run. This will improve reproducibility of model building.

[DirkEilander]

You have to explain this one to me. What would this method do exactly?

In the current setup hydromt maintainers update predefined catalogs and users make their own catalogs. Which catalogs and versions are used is set by the user when building/updating a model (e.g. when initializing DataCatalog, or Model or from CLI). We try to make maintain reproducibility, by writing a catalog of all used sources together with the model files to the model root directory.

[deltamarnix]

Writing a catalog with all used data sources sounds like a good step. I can imagine that we want to update the predefined data catalog every time we run a build, because then we can fix up any compatibility issues with the API of the data catalog. Especially when those data sources are on a network storage and don't contain any versioning. The data catalog has to be repaired.

Would it make sense to host the latest up-to-date data catalog per supported version of HydroMT?
Perhaps we can host a folder, or even do it in gh-pages?

Here is an example on how CMake hosts all its versions. We would have to make a system that generates the predefined catalogs per supported HydroMT version. This would give full flexibility as you mentioned like an API.