Cmorizer as a fix - how to actually fix the variable name ?

[senesis]

Hi everybody. I am a newcomer to ESMValTool, so please forgive any mistake in the method, and shallow knowledge of the tool and doc.

In the ESMValCore doc, there are nice explanations about dataset fix.

The section about common errors claims that

The above example covers one of the most common cases: variables / coordinates that have names that do not match the expected

However, the case of wrong variable name is not actually presented, and I would like to handle it.

What is not clear for me is :

• should the fix method names (which should be variable names) be given the CMOR-standard name or the (wrong) original variable name (the one used in data files, as NetCDF variable name) ?
• when ESMVallTool looks for input files for a recipe, it will use the (CMOR standard) variable name quoted in the recipe for computing the filename, using the relevant 'input_file' entry for the project in config-developper, based on the relevant 'drs' entry in config_user; but in the case of wrong variable name, he should rather use the 'wrong' name for computing the filename. How/where can we define the correspondance between the CMOR-standard variable name and the actual variable name to use for the project, drs and dataset ?

[senesis]

I discovered that Issue #1249 does address a nearby but different concern regarding finding variables with 'wrong' name, and @valeriupredoi stated there (back in aug 2019):

the inter-coupling of variables is something I have started working on but it's not straightforward to implement just yet

I also noticed that maybe there was some progress on that topic, because there is a file esmvalcore/cmor/variables_alt_names with that doc :

This file contains the list of variable short name aliases that are used
in different projects. Will allow us to keep track of changes in variable
short names across projects to simplify the usage for the users

but was not able to find more explanations about the use of that feature for actually handling the case where in a given project, e.g. variable 'tas' is named 't2m'

[senesis]

I also discovered that #950 should give at least part of the answer but :

• is there a HOWTO somewhere (searching doc for 'alias' and 'alternate' wasn't helpful)
• because recipes are implicilty supposed to use CMIP6 Data Request short names (or any other standard, but a single one), wouldn'it it make sense that the aliasing mechanism is project specific (e.g. the config-developper file states for each project which is the project's variable name for each relevant 'standard' variable name)

[bsolino]

Hi @senesis, welcome to the ESMValTool!

I'm fairly new myself, but I've also worked a bit on a cmorizer as a fix that renames variables and I may be able to provide some help (although I'm not sure if it's there are better ways to do it).

Something you can try is to use the AllVars class. This one receives all the cubes and not only the ones for the specific variable. Here you can loop through the cubes looking for the ones that have the wrong name and substitute the right name. You can look at the very early prototype I was doing here: https://github.com/ESMValGroup/ESMValCore/blob/86d170ba74e74a05ef482958af9ed87430af71bc/esmvalcore/cmor/_fixes/native6/emac.py#L25-L29

I'm not sure if that's your specific case, so you can see another kind of renaming in this other fix: https://github.com/ESMValGroup/ESMValCore/blob/master/esmvalcore/cmor/_fixes/obs4mips/ssmi.py

Hope this helps!

[senesis]

Hi @bsolino

Thanks a lot for your suggestions.

However, I am in the case where the dataset file names are formed using the 'wrong' variable name, which means that I must somehow tell the data_finder what this name is. Your examples do not address this issue, but I discovered a way which I describe in next reply

[bsolino]

If the issue is with the data_finder, are you following the naming pattern for native6 in config-developer.yml (https://github.com/ESMValGroup/ESMValCore/blob/622366123bc57df1796a558dec7b24316c113041/esmvalcore/config-developer.yml#L194)? That is the pattern the data_finder uses to locate the files for the appropriate variable. The variable name that you use in the recipe is used to find that variable. I think the ERA5 cmorizer does that: https://github.com/ESMValGroup/ESMValCore/blob/622366123bc57df1796a558dec7b24316c113041/esmvalcore/cmor/_fixes/native6/era5.py#L368-L388

However, following the code by itself may be a bit confusing, I think that maybe @Peter9192 can provide you with more information?

[senesis]

Hi @bsolino

Thanks for your follow-on. My topic is to use ESMVaTool for processing native model data: I do not want to have to move, rename nor pre-process my datafiles upstream of the tool. So I have to create my own filename pattern and declare it in the config file. The issue is that the model output filenames include a 'wrong' variable name, so I have to find a way for the data_finder to use my 'wrong' variable names when replacing {short_name} in that pattern. Besides the way I explain below, I found an alternate way, which implies a small code change in the data_finder, and which I will later describe in an issue

Anyway, thanks again

[senesis]

Digging hard first in the doc, then in the code, I discovered that :

• the name of the CMOR table entry for the variable is used for building the input data filename (i.e. replacing the variable 'short_name' in the DRS patterns)
• that table entry is sought using an 'alternate variables names' feature, which is driven by file esmvalcore/cmor/variable_alt_names.yml
• and, by the way, that the standard data fixes are quite insensitive to the NetCDF variable name, and will anyway rename the single file variable to the 'right' name

This means that I can solve my problem by :

• setting a new project definition which refers to cmor_type e.g. CMIP6
• creating a specific CMOR table, which I derive from a CMIP6 CMOR table and store in the CMIP6 tables directory using an alternate table name prefix (which I set in the new project)
• extending the variable_alt_names.yml file with the needed alias, in order that the table entry with the 'wrong' name is duly found

Elaborating that solution was a bit tedious. I would be glad if an EsmValCore expert could validate that this is the most sensible way to achieve my goal ; this could help other users

Maybe would it be safer to define another way than extending file esmvalcore/cmor/variable_alt_names.yml ? AFAIU, that alias mechanism is rather for define alias across cmor_types, not for aliasing across projects; so I think we could e.g. configuring an aliases dict :

• stored in the table directory associated with the cmor_type used (here CMIP6)
• under name aliases_<new_project>.json
• which keys would be 'right' variable names
• and values would be the 'wrong' variable names
  This would allow recipes written using CMIP6 variable names to work with data from this new project using its own variable names

(this doesn't answer the specific question above about how to name the fix methods)

[bsolino]

I'm not sure touching the CMOR tables is the right approach. If I understood it correctly, they are there to check that the fixed infomation conforms to CMOR standards and modifying them will make you deviate from the standard.

[Peter9192]

Hi @senesis that's some great digging! Maybe I can elaborate a bit. So there's 2 things here,

• The variable name: after the fix it should adhere to the cmor standard. Like you mention, the core will anyway rename the single file variable to the 'right' name ('right' in this case is means following the CMOR standards, so I think that's fair to say).
• The filename: ideally files are stored using a default data reference syntax (DRS). The native6 project allows for custom filenames with the following drs:

input_dir:
  default: 'Tier{tier}/{dataset}/{latestversion}/{frequency}/{short_name}'
input_file:
  default: '*.nc'

So in this case the files should be stored following the input_dir folder structure (which should use the CMOR names).

If you want to support yet another filenaming convention, the easiest way is probably to add that DRS to the config-developer file. If it is variable-specific, you can add additional information to the recipe, like we initially did for ERA5. See e.g.:

• the tags era5_name and era5_freq that are still listed, but no longer necessary, in https://github.com/ESMValGroup/ESMValTool/blob/master/esmvaltool/recipes/cmorizers/recipe_era5.yml and
• https://github.com/ESMValGroup/ESMValCore/pull/645/files, where we switched from using custom filenames to the more generic native6 folder structure).

However, we encourage adhering to existing convention rather than adding custom ones. So if for some reason the existing solutions don't fit for you, we're happy to think along about how to best support your specific use case. In that case, please provide some more information about the specific file naming system used for your data.

Hope that helps.

[senesis]

Hi @Peter9192

Thanks for this explanation. Your solution does not apply to my use case, because my goals are :

• to use model output in their native format, without renaming the datafiles
• to use any existing ESMValTool recipe for processing those model outputs, besides e.g. CMIP6 output, just by adding the corresponding dataset entries ; this means that the trick of 'era5_name' wouldn't work

Hi just wrote an issue for ESMValCore, calling for reactions on a core code change proposal which is quite simple, and would allow to describe the aliasing scheme in the config file, at the level of the project.

[senesis]

Just above, I only adressed the question of filenames . I agree that the core does a good job renaming the variable to the CMOR name

[Peter9192]

Okay I see. Let's continue the conversation in the issue you created 👍

[valeriupredoi]

hello @senesis and cheers for using our Discussions forum! If you need to use a custom variable as it is native to the model you are looking to analyze but not native to CMOR, then I would look at the custom CMOR tables and see if your variable(s) are there already - if not, then you can prepare one or more of those tables for your variables (and create derivation scripts, if the variable can be derived from CMOR variables, but I believe that's not the case for you); altering the cmip5/6 CMOR tables is not an option because we mirror those from PCMDI/CMOR releases and we don't diverge from those, unfortunately standards are there to be uphold. Once you've written a custom CMOR table for each of your variables then you can use them with an existing project (either OBS/obs4mips/ana4mips/CORDEX etc) or if your project is new to our config-developer file, then we'd have to add it there, including all required DRS path templates and file naming templates (or you'd do that by opening a Pull request in ESMValCore). Having said all that, could you please give me a concrete example of your used case? 🍺

[senesis]

I suggest to close this discussion with that summary :

If some input data files, belonging to some weird project having cmor_type=CMIP6, represents, say, some CMIP6 CMOR variables but have their filename formed using another value that the CMOR variable name (let us say, they use t2m instead of tas, hu2m instead of huss ....), and one wants to apply pre-existing recipes on such data (among others) by changing only the datasets section, then there is yet no way except renaming the files, which is not practicable in the case of using ESMValTool in a modeling workflow.

Implementing the change proposed in that issue would allow to describe, at the level of the project declaration, the variable name aliasing scheme to apply to filenames and so to avoid renaming files

[senesis]

ESMValTool Version 2.3 introduced the extra facets, which allow to nicely solve the issue . See an example of its use in this fix code , which uses this config file