ESMValTool workshop Nov 2021

[axel-lauer]

Dear ESMValTool developers,

We are aiming at having another ESMValTool Workshop in fall 2021 and we are trying to find dates that would work for most people. The current idea is to have the workshop in November on three consecutive days, most likely still as a virtual meeting. Please indicate your preferences in the doodle poll here: https://terminplaner4.dfn.de/Xr4dfomBsZypfKRk

If you are interested in attending this workshop, please fill out the doodle at latest by October 24, 2021 (even if none of the dates would work so we can try to find an alternative date).

@ESMValGroup/esmvaltool-developmentteam

[bouweandela]

Topics suggested at the October monthly meeting (#2316):

@rswamina Proposed to have a user engagment team meeting open to anyone interested in user engagement
@alistairsellar Proposed to have a discussion on maintaining recipes
@katjaweigel Proposed to talk about upgrading existing recipes to CMIP6
@zklaus Proposed to have a meeting with the @ESMValGroup/technical-lead-development-team about pinning dependencies
@zklaus Proposed to have a meeting on improving support for regional models
@zklaus Proposed to go over pull requests and seeing if we can get old stuff merged together during the workshop
@valeriupredoi Would like to work on optimization, @zklaus mentioned that improvements for the ocean recipes can be expected with the next iris release. @bouweandela Indicated that he agrees that this is important to keep the tool relevant and will try to work on it in the future.

@alistairsellar asked if the workshop will still be a combination of talking and actually programming things together, @remi-kazeroni confirmed that this will again be the case

[axel-lauer]

Dear @ESMValGroup/esmvaltool-developmentteam,

We now fixed the dates. The next ESMValTool workshop will take place from 23-25 November 2021. Please save the date! We will probably have six sessions in total, a morning session and an afternoon session on each of the three days. A draft agenda and requests for organizing sessions will be sent around soon.

[axel-lauer]

Here some additional workshop topics (in addition to the topics suggested at the October monthly meeting (#2316)) that we could cover. More suggestions are very welcome!

@bouweandela Discussion about asking all members to enable 2-factor authentication. That way, even if a hacker manages to obtain someone's password, they still cannot use it to log in.
@bouweandela It would be good to think about what we do with members who are no longer active and remove those from the GitHub team, for example, if people have not contributed for 2 or 3 years, or we know they have left the project.
@bouweandela @remi-kazeroni @axel-lauer Revisiting the organization of the “core team” to avoid duplicates between full members and child team members. At the moment it is not easy to have a quick overview of who belongs to this team.
@schlunma Recipe/diagnostics for plotting generic output (quicklooks) #2184
@schlunma ICON CMORizing ESMValGroup/ESMValCore#1079
@remi-kazeroni Improving handling and communication about “backward incompatible changes” that make the tool difficult to run and develop.
@ALL Follow up of the recipe testing strategy meeting – maintaining recipes, updated recipes with CMIP6 data

[axel-lauer]

Dear @ESMValGroup/esmvaltool-developmentteam,

From the topics named for the workshop, we would like to propose six sessions for our November 2021 workshop, one morning session and one afternoon session on each of the three days:

1. Developing “scientific” test recipe(s) including time for writing the test recipes and setting up infrastructure
2. Strategy for handling and communicating implementation of non-backward compatible changes (e.g. extending list of tests to perform before merging to minimize unnoticed breaking of recipes by new core changes)
3. New features - presentations:
   a) automatic downloading of files & reusing preprocessor output
   b) using X first/last years
   c) installation using mamba
4. Generic diagnostics and reading / regridding native model input
5. Strategy for improved support of regional models
6. General discussion (handling of members (2-factor authentification + inactive members + communication), error “tolerance” of the tool, teams, recipe maintainers, pull request reviewers, performance optimization of the tool)

In addition, we would like to propose two side meetings:

1. Slot for the User Engagement Team
2. How to pin dependencies

Please let us know what you think. Suggestions and comments highly welcome!

[valeriupredoi]

I probably won't be able to attend much of the workshop, but it has been suggested a few times to work on/discuss optimizing the performance of the tool. Would that be another topic, or is that already discussed elsewhere?

yes 😁

[valeriupredoi]

btw what's those peasantations all about? 😁

[axel-lauer]

I probably won't be able to attend much of the workshop, but it has been suggested a few times to work on/discuss optimizing the performance of the tool. Would that be another topic, or is that already discussed elsewhere?

I'll add "performance optimization" to the general discussion, so we don't forget...

[axel-lauer]

btw what's those peasantations all about? 😁

We thought we might approach our experts on these topics to either give a brief overview with some slides or maybe even a live demo... 😁

[valeriupredoi]

sounds like a solid plan @axel-lauer - lemme know if you want a peasantation from me 😁

[ehogan]

Hi everyone,

With regards to the "Developing scientific test recipe(s)" session, @Jon-Lillis and I have developed a prototype recipe test from code already available in ESMValTool, see our prototype_recipe_test branch.

The prototype test runs the recipe_validation_CMIP6.yml recipe. The test uses dummy data and takes about 70 seconds to run (which is comparible to the normal duration when running this recipe; we are not sure how much of the test timings are due to overhead of the test framework and we are planning to perform profiling to learn more). Note that tests like this prototype test would not pick up the missing data issues found during the recent testing performed for the release, but would enable the recipe to be tested regardless. There may also be recipes that would not work well with dummy data, due to e.g. thresholds. However, it should be possible to run tests like this prototype test on a high fraction of the available recipes to achieve a good recipe test coverage.

To clarify, the purpose of the prototype test is to simply run the recipe to ensure it completes without errors. This will ensure any breaking changes made to ESMValCore and ESMValTool can be discovered easily and quickly. No comparisons of outputs are performed. There are a few hard coded pieces of code that need replacing (I'm sure there will be functions that already exist in ESMValCore / ESMValTool to do what needs to be done in a generic way, but help from someone who knows the code better than us would be most appreciated!); once that is complete, it should be possible to start adding more recipes for testing in this way :)

These tests would complement the tests proposed at the testing workshop.

Would it be possible to set aside some time during the "Developing scientific test recipe(s)" session to discuss / develop this further, please? :)

Paging @bouweandela @alistairsellar for comment :)

[valeriupredoi]

(we can talk about this at the workshop, just some pointers for you guys to think about 😁 )

[bouweandela]

I just remembered I wanted to ask whether you know who was involved in the development of this test framework code before it was abandoned

That would be me, building on the work done by @BenMGeo and @bulli92 before I joined the project, if I remember correctly.

[ehogan]

also, there is the question of OBS data - what are you planning on doing with that - simulating it as well?

Great question! We are also using dummy data for the observational data. Coincidentally the observational data in our prototype recipe have the same shape as the model data, so it just worked, but one thing I hadn't considered is if this wasn't the case. Are the observational data also described in MIP tables?

[ehogan]

There is, however, the question of array sizes that need to match the recipe specs, since some of the diags are meant to use eg at least 10 years of data, otherwise they'd crash with index out of bounds and such guff

We are already using the start and end years from the recipe to contruct the time dimension appropriately :)

[valeriupredoi]

Are the observational data also described in MIP tables?

Unfortunately not, since they come in a garden variety, each with all sorts of metadata and data inconsistencies (vs CMOR, that is, they are perect in the eyes of the OBS data creators tho 😁 ), we have a large set of OBS cmorizers if that would help you in the project, so the OBS data that gets ingested into ESMValTool is CMORized and clean - even cleaner than CMIP data that constantly needs fixes

[axel-lauer]

Hi all @ESMValGroup/esmvaltool-developmentteam,

Here is the final agenda for the next ESMValTool workshop. Please find the zoom link below!

Agenda Technical ESMValTool Workshop 2021-2 (via video call)

(all times are given in CET)

23 November 2021 - Day 1 (Tuesday)

10.15 am – 10.20 am Introduction, welcome and workshop agenda (@hb326)
10.20 am – 10.30 am Update and current status of the consortium agreement (@alistairsellar)
10.30 am – 12.30 pm ESMValTool v2.4 and new features (@zklaus)

1. Introduction to v2.4 (@zklaus)
2. Downloading of missing files & reusing preprocessor output (@valeriupredoi )
3. 🐍 Installation using mamba (@valeriupredoi)
4. Using X first/last years (@sloosvel)

12.30 pm – 01.30 pm Lunch break (opportunity to virtually hang out with fellow workshop participants)
01.30 pm – 03.30 pm Development of scientific test recipe(s) (including writing the test recipes and setting up testing infrastructure) (@axel-lauer, @hb326)

1. Introduction prototype using dummy data (@ehogan)

03.00 am – 04.00 pm Side meeting: User Engagement Team (@rswamina)
03.30 pm – 04.00 pm Coffee break (opportunity to virtually hang out with fellow workshop participants)
04.00 pm – 04.30 pm Summary and wrap up of first day (@hb326)

24 November 2021 - Day 2 (Wednesday)

10.30 am – 11.30 pm Generic diagnostics and reading / regridding of native model output (@schlunma)
11.30 am – 12.30 pm Side meeting: How to pin dependencies (@zklaus)
12.30 pm – 01.30 pm Lunch break (opportunity to virtually hang out with fellow workshop participants)
01.30 pm – 02.30 pm Strategy for improved support of regional models including team formation for coding (@zklaus)
02.30 pm – 03.00 pm Coffee break (opportunity to virtually hang out with fellow workshop participants)
03.00 pm – 04.00 pm Discussion of preliminary test recipes (@hb326, @axel-lauer)
04.00 pm – 04.30 pm Summary and wrap up of second day (@axel-lauer)

25 November 2021 - Day 3 (Thursday)

10.30 am – 12.00 am Strategy for handling and communicating implementation of non-backward compatible changes with the aim of minimizing unnoticed breaking of recipes new changes (@alistairsellar)
12.00 am – 01.00 pm Cross team meeting (@alistairsellar)
01.00 pm – 02.00 pm Lunch break (opportunity to virtually hang out with fellow workshop participants)
02.00 pm – 03.00 pm General discussion (@nielsdrost)

1. Handling of members (2-factor authentication + inactive members + communication)
2. Error “tolerance” of the tool
3. Teams, recipe maintainers, reviewers
4. Optimization of the tool’s performance

02.30 pm – 03.00 pm Coffee break (opportunity to virtually hang out with fellow workshop participants)
03.00 pm – 04.00 pm General discussion - continued (@nielsdrost)
04.00 pm – 04.30 pm Wrap up (@alistairsellar, @hb326)

Gitter

https://gitter.im/ESMValGroup/Lobby?utm_source=sharelink&utm_medium=link&utm_campaign=share-link#

Link for video call

https://uni-bremen.zoom.us/j/95461223745?pwd=NkZBSmpGMmFyZnFtMWlNcmlhRmlDQT09

[bouweandela]

Thank you for creating the agenda! I would be interested in joining the meetings on testing, because this is a topic that I plan to work on next year as part of the IS-ENES3 project. Unfortunately, I am only available on Thursday (I may be able to join at most 1 hour on Tuesday, but not at all on Wednesday). Would it be possible to do a short summary of the recipe testing meetings as part of the general discussions taking place on Thursday?

[valeriupredoi]

@axel-lauer sweet agenda 🍺 I've added two active links pointing to tomorrow morning's guff from me ☕

[valeriupredoi]

@ledm FYI agenda, bud

[katjaweigel]

The recording is available for 30 days at: https://seafile.zfn.uni-bremen.de/d/3d7b594738844b468ba1/
I've set a password protection (I tell you at the meeting), feel free to share it with people you know but I didn't want to make it completely public.

[alistairsellar]

Backward compatibility discussion

Hi everyone. We have a discussion tomorrow morning on backward compatibility.

The background here is that some users have complained about their recipes or diagnostics breaking when they upgrade to a new version. So the goal is to have a discussion to define the issue more precisely, and begin to discuss ways forward. Some questions to think about:

• Should we be looking to update the backward compatibility policy? Does the policy deliver what users and developers need?
• Do we need to better communicate about the policy and about individual changes?
• What support do users need in coping with backward-incompatible changes?

If you have time and interest, here is some background pre-reading that could be useful context...

• The current backward compatibility guidance for ESMValTool developers: https://docs.esmvaltool.org/projects/ESMValCore/en/latest/contributing.html?highlight=version#backward-compatibility
• Semantic versioning will likely come up in the discussion, so it would be worth reading this page for what I believe is the canonical interpretation of semantic versioning: https://semver.org/
• And riding my usual hobby horse about not re-inventing the wheel, those who have a keen interest might enjoy this philosophical document that explains NumPy's principles on the topic: https://numpy.org/neps/nep-0023-backwards-compatibility.html (Feel free to add links to other projects that you think are relevant.)

[bouweandela]

From previous discussions on this topic, my impression is that the people who run into problems most are those who are using the ESMValCore installed from source, when it would have been a better choice to use the released version.

Cases where using the released version is the best option:

• This should be the default choice for everyone. ESMValCore is under active development and it is unavoidable that occasional mistakes are made in during that process. These have usually been noticed and fixed by the time the ESMValCore package is released.
• If you are working on a recipe/diagnostic for several months
• If you are not able to go back to the previous version of ESMValCore you had installed after running git pull in your copy of the ESMValCore repository (in this case it might be a good idea to talk to a more technical colleague or someone from the @ESMValGroup/tech-reviewers team to give you some hints on using git).

Cases where using installing ESMValCore from source is the best option:

• You are developing a new feature (preprocessor, fix, etc) for ESMValCore
• You really need the very latest ESMValCore features

There is already a warning about using the development version of ESMValCore instead of the latest released version: https://docs.esmvaltool.org/en/latest/quickstart/installation.html#using-the-development-version-of-the-esmvalcore-package, but maybe it could be made more clear.

[bouweandela]

Looking at the release notes for v2.4, I realized that the backwards-incompatible changes were hard to find. To improve this, I made a pull request (#2431) with an update to the script that generates these notes from the pull request titles, so they will listed in a separate section at the top.

[valeriupredoi]

cheers Bouwe! I approved it so it's ready to go, I don't think we have any in 2.4. BTW you can search for a label using this in the search field is:closed label:"backwards incompatible change" - and then either go through the list or click on the label and that'll show you all the issues/PRs with that ;label 👍

[bouweandela]

BTW you can search for a label using this in the search field is:closed label:"backwards incompatible change" - and then either go through the list or click on the label and that'll show you all the issues/PRs with that ;label +1

Thanks V, I did just that and found 10 backwards-incompatble changes in ESMValCore v2.4. I think it is important to highlight those in the release notes.

[valeriupredoi]

oh yeah forgot to say I didn't look in Core 😁

[bouweandela]

Development of scientific test recipe(s)

@axel-lauer @hb326 Did the meetings you had on developing scientific test recipes result in some meeting notes or plans? Would you be able to share those, e.g. in an issue or discussion?

[axel-lauer]

Some preliminary draft notes from the session on scientific test recipes:

• Simulated data and testing framework introduced (Emma Hogan)
• intermediate strategy for testing until a more comprehensive testing is available: small testing recipes working with a reduced set of real data and commonly used operations
• Examples developed during the workshop include
  • https://github.com/ESMValGroup/ESMValTool/tree/workshop_test_recipes_al
  • Testing CMIP3, 5, 6 anomalies, horizontal maps, perfmetrics: https://github.com/ESMValGroup/ESMValTool/tree/test_recipe_lisa
  • Reduced version of existing recipe: https://github.com/ESMValGroup/ESMValTool/tree/fast_version_deangelis_for_test
  • https://github.com/ESMValGroup/ESMValTool/tree/recipe_eyring13jgr_fig12_testrecipe
  • https://github.com/ESMValGroup/ESMValTool/tree/Arctic_telecon
• Intermediate solution, complimentary to simulated data approach until a more sophisticated testing framework is in place
• Proposal to approach individual developers to provide a testing recipe from their own science recipes
• Testing and reporting framework needed to run the testing recipes (use ESMValBot?)
• Report progress here: [recipe testing strategy] Test combinations of preprocessor functions ESMValCore#1361

[bouweandela]

Thanks for the updates! This seems to go in a somewhat different direction than what was discussed at the testing workshop #2259?

from #2346:

Could we develop synthetic data on which all recipes would run? Consensus was reached that developing such dataset would be difficult as ESMValTool runs on a great variety of data. These data may also not capture the trends and thresholds analyzed in recipes.

from the post above:

Simulated data and testing framework introduced (Emma Hogan)

Does this mean that there is no consensus or did the people in the meeting change their mind on this topic?

from #2346:

Would it help to have a "test version" of each/many recipes that could be computationally less demanding, possibly using less datasets and a narrower time range? That would imply twice more recipes to maintain.

from the post above:

Proposal to approach individual developers to provide a testing recipe from their own science recipes

Does this mean that we would like to maintain two recipes for every recipe after all?

[ehogan]

Could we develop synthetic data on which all recipes would run? Consensus was reached that developing such dataset would be difficult as ESMValTool runs on a great variety of data. These data may also not capture the trends and thresholds analyzed in recipes.

from the post above:

Simulated data and testing framework introduced (Emma Hogan)

Does this mean that there is no consensus or did the people in the meeting change their mind on this topic?

Hi @bouweandela! The prototype tests we are developing use dummy data (ones) rather than simulated data. We accept this will not work for all recipes, but we hope this will allow a large fraction of the recipes to be run quickly to ensure the recipes can run without errors. It may be possible to develop a "simulated data" option in the future, but it is not necessary for the prototype tests we are developing.

[ehogan]

from #2346:

Would it help to have a "test version" of each/many recipes that could be computationally less demanding, possibly using less datasets and a narrower time range? That would imply twice more recipes to maintain.

from the post above:

Proposal to approach individual developers to provide a testing recipe from their own science recipes

Does this mean that we would like to maintain two recipes for every recipe after all?

This was one of the reasons we developed the prototype tests, since this would allow recipes to be tested without duplicating recipe files, which would create additional maintenance effort to keep them aligned with the original recipe files.

[katjaweigel]

from #2346:

Would it help to have a "test version" of each/many recipes that could be computationally less demanding, possibly using less datasets and a narrower time range? That would imply twice more recipes to maintain.

from the post above:

Proposal to approach individual developers to provide a testing recipe from their own science recipes

Does this mean that we would like to maintain two recipes for every recipe after all?

I think the idea is not to have a test version of every recipe, but only of a hand full, which could be easily run after changes. They should cover some typical "real world" cases hoping to quickly detect relevant effects.
So far this and the synthetic data are independent ideas to improve the testing, but maybe this could be combined at some point to be independent from real data.