Caching results : rationale, scope, method, and possible work-around

[senesis]

Caching results could be much useful for compute efficiency. This has been experimented in CLiMAF. The attached document Caching_EVT_results.pdf explains how it is done in CLiMAF and drafts how it could be implemented in ESMValTool.

In email discussions :

@zklaus reported that :

The main reason why we don't have [ a cache ], and indeed why the closest thing that we had to it was removed in going from version 1 of the tool to version 2, is the reproducibility of results.
Particularly Bouwe argued that we would need to track all (ABI?) versions of all involved dependencies to make this feature reliable.

while @valeriupredoi noted that :

the problem of dependency versioning is a very sensitive one wrt reproducing and caching. However, I can see a solution for that using conda lock files and frozen environments built with them. This would open another can of worms though - that of folding in the latest releases with the latest developments

@Bouwe added :

Note that since the latest version of the tool we have (limited) support for re-using preprocessed data

The reproducibility issue raises a question : what should be the scope of cache use ?

The actual experience with CliMAF is that the cache is mainly used on an individual basis, in the course of developing and refining an analysis, and not as a long-term archival of reproducible reference results. This means that cached results are useful on a timescale which is not prone to see much change of the dependencies, and that it could make sense to adopt such a limited scope for an ESMValTool cache. In such a case, this should of course be made very clear for users.

Additionally, reproducibility of ESMValTool results should rather rely on provenance information, which is best suited for hosting quite extensive dependencies description

The resume feature included from version 2.4 brings some progress for re-using previous results : it allows to re-use the output of the successful post-processing stages from a previous run, on a diagnostic+variable basis. Its limitations are, AFAIU :

1. the user has to diagnose by himself if the list of datasets and the parameters of the pre-processing are the same for the current run and the previous run
2. successful scripts are re-run anyway
3. when wishing to add a dataset, there is is no way to re-use the pre-processing results for the other datasets
4. for a script which has an explicit ancestor task, there is no way to re-use previous ancestor results
5. the user interface misses a handy auto-resume option, by which ESMValTool would re-use all run directories for the same recipe name in the same output_dir

Dropping some of these limitations could be a good compromise for improving the situation

[senesis]

5. the user interface misses a handy auto-resume option, by which ESMValTool would re-use all run directories for the same recipe name in the same output_dir

An alternate way would be that the same run directory can optionally be used again and again until successful run (thus avoiding to copy or link results from previous run between run directories)

[senesis]

@zklaus , @valeriupredoi , @bouweandela
Our email discussion on that subject is now turned to a formal discussion here. Given the further proposal for a workaround here above, how would you prefer to carry on : teleconf or contributions here ?
( @jservonnat is also interested)