New fesom2-recom version in the esm tools on levante

[mandresm]

Sent by @mseifert93

Hi Miguel,

I have two small question about the esm tools, and I would be super grateful if you could help me...

I have a modified ocean-only fesom-recom code and want to do spin-ups on levante using the esm-tools. The git repository of > this new code is here:
https://github.com/FESOM/fesom2/tree/fesom2_3zoo_ballast

The tricky part of this code is that it has a flexible setting depending on the number of plankton groups (we have four different versions in total) - Özgür implemented this by adding a few lines to the src/CMakeList which define if the entire code is compiled (i.e., all plankton groups are used) or of only a part of it is compiled (excluding code parts that are flagged with __use_coccos, for example). For example:
target_compile_definitions(${PROJECT_NAME} PRIVATE __recom USE_PRECISION=2 __coccos __3Zoo2Det)

In my local esm tools version, I added the new fesom2 branch to the fesom.yaml:
/home/a/a270120/esm_tools/configs/components/fesom/fesom-2.1.yaml
I added it 4 times pointing to different config/namelists, depending on the version that I want to use. Maybe not the smartest way, but as long as it works it's fine :-)

I managed to install the model four times, I adapted the CMakeLists accordingly, and recompiled each of the models:
/home/a/a270120/fesom2_recom_models

Theoretically I could start test simulations now. I can, however, not find a suitable runscript in the folder with example runscripts - they are all for ollie, mistral or include medusa. Do you have a suggestion what to use in my case?

And my second question is, of course I will have a different namelist.io.recom (still need to adapt those in the config file of each model - I know, again not the cleanest way, but I'm just happy if it works) than what was probably used in earlier versions of the ocean-only code. Do I have to change anything in the esm tools? Or does it only do what is given in namelist.io.recom in model_code/config? I remember that I had to change recom.yaml and/or recom_output.yaml when I was adding diagnostics to AWI-ESM, but maybe it is different for ocean-only?

Thanks a lot in advance!!

I may come back to you as I also want to use a different forcing and probably need to transfer the forcing files from albedo to levante. But I will first discuss this with Judith and Özgür.

Cheers,
Miriam

[pgierz]

Internal Comment for esm-tools devs: Mmmmm....I do not like compile time changes affecting how runtime can work. If we have time maybe we can find a cleaner way of doing this.

For @mseifert93:

Theoretically I could start test simulations now. I can, however, not find a suitable runscript in the folder with example runscripts - they are all for ollie, mistral or include medusa. Do you have a suggestion what to use in my case?

We can help you adapt one of the other runscripts, probably that is best done via a video meeting and one of us can tell you what to write down. Then we can share the resulting run configuration.

, of course I will have a different namelist.io.recom ...skipped... Do I have to change anything in the esm tools?

Nope! :-) That's what namelist.changes are for. Have a look in the handbook: https://esm-tools.readthedocs.io/en/latest/cookbook.html#changing-namelist-entries-from-the-runscript

[mandresm]

Hi @mseifert93,

I'm tagging @chrisdane because I am aware he's been working in some of these things so we can avoid duplication of work.

In my local esm tools version, I added the new fesom2 branch to the fesom.yaml:
/home/a/a270120/esm_tools/configs/components/fesom/fesom-2.1.yaml
I added it 4 times pointing to different config/namelists, depending on the version that I want to use. Maybe not the smartest way, but as long as it works it's fine :-)

Can you push that local branch so that I can see exactly what's going on? I would always recommend to push to GitHub and commit often, then you have a backup, a history of your work and you can easily share it with others :)

Theoretically I could start test simulations now. I can, however, not find a suitable runscript in the folder with example runscripts - they are all for ollie, mistral or include medusa. Do you have a suggestion what to use in my case?

To add to what @pgierz said, the runscripts there are written from the users to the users, and many of them are outdated. As @pgierz, please try to adapt one of them, I would recommend one of the levante ones, for example https://github.com/esm-tools/esm_tools/blob/release/runscripts/fesom2/fesom2.1-initial-monthly.yaml

If you encounter any problems please let us know, if it gets too complicated we can schedule a meeting with you.

(still need to adapt those in the config file of each model - I know, again not the cleanest way, but I'm just happy if it works)

This is a good approach for prototyping, but I would encourage you that once it works we sit together, cleanup and merge to main.

And my second question is, of course I will have a different namelist.io.recom (still need to adapt those in the config file of each model - I know, again not the cleanest way, but I'm just happy if it works) than what was probably used in earlier versions of the ocean-only code. Do I have to change anything in the esm tools? Or does it only do what is given in namelist.io.recom in model_code/config? I remember that I had to change recom.yaml and/or recom_output.yaml when I was adding diagnostics to AWI-ESM, but maybe it is different for ocean-only?

As @pgierz mention, you could modify the namelist with namelist_changes from your runscript. However, namelist.io is for the most part a list of variables, and a long one, so it really doesn't make sense to put it on your runscript, specially if you have for example 4 different namelist.io that you are always going to be using. In that case, I suggest you wear your "developer hat" (you're not a user anymore) and create 4 different namelist.io.recom somewhere and use choose_ blocks, config_source and varaible substitutions (${}) to generalize the problem.

If you push and share your branch, and pull-request as draft to release we can help you much faster without even needing to schedule a meeting :) This way we can give you also inline advise to get a clean and functioning configs really fast

[mandresm]

However, I still have an issue with the namelists, it does not find namelist.oce - would it be possible for you to check what I missed to modify?

From this line in your branch:

https://github.com/mseifert93/esm_tools_fesom2.1-recom/blob/239f334716b04a9e5092a7e35b6b2179017ca439/configs/components/fesom/fesom-2.1.yaml#L631

do you have a oce: "${namelist_dir}/namelist.oce.recom" file in the namelist_dir? And what is the actual value of your namelist dir? (you can check it on the *_finished_config.yaml in your experiment <path>/<expid>/run_DATE/config/*_finished_config.yaml).

[mseifert93]

I define the namelist_dir in the fesom-2.1.yaml (https://github.com/mseifert93/esm_tools_fesom2.1-recom/blob/esm-tools_fesom2.1-recom_4versions/configs/components/fesom/fesom-2.1.yaml)
namelist_dir: "${model_dir}/config/case_3p3z2d/"
and I have a namelist.oce in this directory. The directory seems to be identified correctly.
But you are right, further down I have this line:
oce: "${namelist_dir}/namelist.oce.recom"
which would need to be modified to
oce: "${namelist_dir}/namelist.oce"
correct? This modification would be fine with the esm tools (for fesom-recom it is, because namelist.oce is our standard name), right?

[mandresm]

correct? This modification would be fine with the esm tools (for fesom-recom it is, because namelist.oce is our standard name), right?

This will break things for the work of Ying, if I remember correctly. The solutions are trivial, you just need to decide how you want to not conflict with Ying's work, the design choice is yours and Ying's.

[mseifert93]

Yes, I know - Ying, Chris and me have to sit down at some point to make things compatible for both code versions. But at the moment I urgently need a working version for my code.
Next problem is that it obviously recognizes my namelist.io.recom (the list of output variables, this one is different than the recom output list in the esm tools which is still for the old fesom1.4 code version), but it doesn't use it. As a result, the output files and the restart files are not written and it breaks at the end of the first month (I use monthly restarts for now). Why doesn't it actually use the namelist.io.recom? I cannot modify the recom output list in the esm tools, as namelist.io.recom is different for each code version. It would be necessary to be able to use the one which is in the code-config folder.

[pgierz]

Hi @mseifert93,

hmm...maybe the FESOM code (in the FORTRAN world) is hard-coded to use namelist.io instead of namelist.io.recom? I did not look at the code yet, that is just a speculation.

Here are some next steps I would do:

• Do a ls in your work folder after submitting with the -c flag. Name sure that your nameless there are named correctly and have the contents you expect. That would eliminate esm-tools being badly configured and instead we will need to look into the FORTRAN part...
• It looks like there is a fesom.namelist_dir defined here as well. Check that the right one wins. If needed, you can overwrite it again from your user runscript.
• Check that the model wrote all the outputs you expect in the work folder before esm-tools cleans up. The files might be there, but might not be copied/moved/linked back to the main experiment location.
• You can use add_ in to change the restart files esm-tools knows about. Those are defined here. From what I can see, there is nothing recom specific there. I can give you a more detailed syntax explanation if you need, but just follow some of the other examples.

Let me know if that helps! 🙂
Paul

[chrisdane]

Hi all

Sorry I was on holiday.

Please see the awicm3_recom branch where I tried to add awicm3+recom support (i.e. coupled) as well as fesom2+recom (i.e. ocean-only). There, the requirements listed by @mseifert93 are already included, e.g. adding namelist.io.recom (should not interfere with awi-esm-2-recom stuff) and having different compiler options depending on the specified model version (2p1z1d, 3p1z1d, ...).

Focusing the discussion here on fesom2+recom, the problem for us that it is not clear if fesom2+recom is a component or a setup. Following suggestions from @pgierz (on webex), we decided to try the component approach by including this (https://github.com/esm-tools/esm_tools/blob/awicm3_recom/configs/components/fesom/fesom-2.1.yaml):

configs/components/fesom/fesom-2.1.yaml
choose_fesom.with_recom:
    True:
        ...
        # compiling options of recom-3.0
        choose_fesom.version:
            "2.1-recom-3.0":
                add_further_reading: recom.yaml
                recom:
                    version: "3.0"
                ...

However, this does not work due to

RecursionError: maximum recursion depth exceeded while calling a Python object

As I understood @pgierz they wanted to come up with a solution for this?

[pgierz]

I wanted to make a fix for this, yes, but I need to discuss with @mandresm how best to do it. From my perspective, RECOM can be best described as an "embedded component", with the best comparison being to how JSBACH is "embedded" into ECHAM6.

I had tried to pull apart the configuration into more easily understandable units, but this quickly got out of hand.

In effect the problem is the following:

It boils down to the fact that an sed replacement is done at compile-time to the CMakeLists.txt, based upon a variable that is not resolved by esm-master. For historical reasons, not all variables inside the configuration are resolved at ...emm.... esm-master-time (I hate that I just invented this expression). Debugging this with inside knowledge is difficult enough, and without knowing how esm-master works it is impossible. I'll have some time during this week to look into it in more detail again.

I am maybe missing some key information here, but why does this need to be a compile-time choice? Depending on which combination of zooplankton/detritus you use, can't that just be a namelist setting? If that is possible it would make this whole problem go away...

[chrisdane]

Thanks for the reply!

Problem 1: component or setup
Would it be possible for you to find a provisional solution for the maximum recursion depth error above so that we can continue working? We need to start a fesom2+recom spinup as soon as possible and it would be very sad to do it - again - without the esm_tools :/

Problem 2: sed for compiling
I agree, setting the recom variant (2p1z1d, 3p1z1d, ...) via namelist settings would be much better than using compiling flags. I assume, however, changing this in the code would involve more work than implementing this case into the esm_tools since its just another sed command? Can you please give a recommendation which solution would be faster in your view? Then we can discuss it with the recom people.

Thanks a lot!

[mandresm]

Problem 1: component or setup
Would it be possible for you to find a provisional solution for the maximum recursion depth error above so that we can continue working? We need to start a fesom2+recom spinup as soon as possible and it would be very sad to do it - again - without the esm_tools :/

yes, @pgierz do you know how to reproduce this? if you know, we can sit together tomorrow and have a look at the problem

Problem 2: sed for compiling
I agree, setting the recom variant (2p1z1d, 3p1z1d, ...) via namelist settings would be much better than using compiling flags. I assume, however, changing this in the code would involve more work than implementing this case into the esm_tools since its just another sed command? Can you please give a recommendation which solution would be faster in your view? Then we can discuss it with the recom people.

This depends on how complicated would it be to do it in FESOM. In ESM-Tools should be pretty fast, but then again we are patching with ESM-Tools a bad practice in the models... Also, it's a very bad idea to have to use different binaries for the same version of the model and different number of tracers. In any case, you guys might have very good reasons for why are you doing it this way, and we might be not having the whole context, so we will help with whatever decision you make

[mandresm]

+1 to component + subcomponent, no need to use a fesom-recom setup

[pgierz]

Yeah, I know where it is. Just come by my office tomorrow after the GM when you have time. I'll pull up my reproducible example. I think it's actually easy enough, I was just tired and started only seeing fuzzy colorful characters and numbers instead of actual code...