How to configure when adding support for new native datasets?

[stacyqxz]

Hello everybody. I am a newcomer to the ESMValTool, so please forgive me for asking dumb questions.

I have a locally folder storing reanalyzed cmip6 monthly datasets (over 100 of them), and I want to use ESMValTool to analyze them in a fast manner. In the ESMValCore doc, there is a short section covering how to locate data locally. But it doesn't talk much about a method that can combine local datasets and cmip6 cmor-drs.

My first thought is to add a new key representing the data organization (as suggested in the doc). For the input_dir, I put the directory to which the datasets are stored. Where I am stuck is the input_file. I feel the placeholders should be compliant to cmip6 keys for some reason but don't know why. And I couldn't find a document that lists the keys. For example, the variable stored in my datasets is tasmin. Should I call it var or variable_short_name or something else?

I don't know if I state my question clearly. I would really appreciate if someone can point me to some documentations and references, or answer this question.

[stacyqxz]

Is this CMIP6 Data Request page the right document?

[zklaus]

Welcome, @stacyqxz, it's nice to have you here.

The data request that you link does contain a lot of the information that you might need, but to me it is not clear right now what you mean by

reanalyzed cmip6 monthly datasets

Could you be a bit more explicit? For example, tasmin is published as part of the CMIP6 data. Have you processed this further? Does the result still follow the general format of the CMIP6 data?

[stacyqxz]

Thank you for your reply! Sorry for not being clear enough. The reanalysis data I downloaded are the BCCAQv2 downscaled CMIP6 datasets and they follow the general format of the CMIP6 data.

I think I managed to configure both config-developer.yml and config-user.yml files given the fact that the datasets can be found in the defined dir. The next issue I encountered is an esmvalcore.cmor.check.CMORCheckError. For pr, the current unit is mm day-1 and it says the unit cannot be converted to kg m-2 s-1. For tasmin, the error says 'there were errors in variable tasmin: height2m: does not exist'.

Any idea how I can fix this? For pr, my thought is to fix megadata following this page. Not sure about it tho.

[stacyqxz]

Updates:

• I was able to find the datasets from the directory with key-value pairs.
• Found two discrepancies between my downscaled datasets and the CMIP6 format, one for pr and the other for tasmin (so probably for tasmax as well).

1. The unit for pr in my datasets is mm/day and this raises a esmvalcore.cmor.check.CMORCheckError saying that it cannot be converted to kg m-1 s-1. However, I don't really want to convert it to kg m-1 s-1. Is there a way to keep mm/day without raising errors?
2. The esmvalcore.cmor.check.CMORCheckError for tasmin is missing height2m. The downscaled datasets are about monthly tasmin/tasmax derived from CMIP6 data. So it should still be height2m but it's missed in the metadata (my guess, not sure tho). Is there a way I can add this info or fix this issue?

[stacyqxz]

The output of ncdump -h MYFILE.nc

netcdf tasmin_monthly_average_BCCAQ2v2+ANUSPLIN300_CanESM5_historical+ssp585_r1i1p2f1_1950-2100 {
dimensions:
        lon = 1068 ;
        lat = 510 ;
        time = 1812 ;
variables:
        double lon(lon) ;
                lon:units = "degrees_east" ;
                lon:long_name = "lon" ;
                lon:standard_name = "longitude" ;
                lon:axis = "X" ;
        double lat(lat) ;
                lat:units = "degrees_north" ;
                lat:long_name = "lat" ;
                lat:standard_name = "latitude" ;
                lat:axis = "Y" ;
        double time(time) ;
                time:units = "days since 1850-01-01 0:0:0.0" ;
                time:long_name = "Time" ;
                time:calendar = "365_day" ;
                time:standard_name = "Time" ;
        float tasmin(time, lat, lon) ;
                tasmin:units = "degC" ;
                tasmin:_FillValue = -32768. ;
                tasmin:standard_name = "Monthly Average Minimum Temperature" ;
                tasmin:long_name = "Monthly Average Minimum Temperature" ;

// global attributes:
                :institution = "Pacific Climate Impacts Consortium (PCIC), Victoria, BC, www.pacificclimate.org" ;
                :contact = "Pacific Climate Impacts Consortium" ;
                :Conventions = "CF-1.7 CMIP-6.2" ;
                :institute_id = "PCIC" ;
                :domain = "Canada" ;
                :creation_date = "2019-10-21T14:13:53PDT" ;
                :frequency = "day" ;
                :product = "downscaled-output" ;
                :mip_era = "CMIP6" ;
                :modeling_realm = "atmos" ;
                :activity_id = "CMIP" ;
                :table_id = "day" ;
                :variable_id = "tasmin" ;
                :references = "Alex J. Cannon, Stephen R. Sobie, and Trevor Q. Murdock, 2015: Bias Correction of GCM Precipitation by Quantile Mapping: How Well Do Methods Preserve Changes in Quantiles and Extremes?. J. Climate, 28, 6938–6959." ;
                :downscaling_method = "Quantile Delta Mapping" ;
                :downscaling_method_id = "BCCAQv2" ;
                :downscaling_package_id = "github.com/pacificclimate/ClimDown" ;
                :driving_branch_method = "Spin-up documentation" ;
                :driving_data_specs_version = "01.00.29" ;
                :driving_experiment = "historical,ssp585" ;
                :driving_experiment_id = "historical,ssp585" ;
                :driving_forcing_index = "1" ;
                :driving_grid = "T63L49 native atmosphere, T63 Linear Gaussian Grid; 128 x 64 longitude/latitude; 49 levels; top level 1 hPa" ;
                :driving_grid_label = "gn" ;
                :driving_further_url_info = "https://furtherinfo.es-doc.org/CMIP6.CCCma.CanESM5.historical.none.r1i1p2f1" ;
                :driving_institution = "Canadian Centre for Climate Modelling and Analysis, Environment and Climate Change Canada, Victoria, BC V8P 5C2, Canada" ;
                :driving_institute_id = "CCCma" ;
                :driving_model_id = "CanESM5" ;
                :driving_nominal_resolution = "500 km" ;
                :driving_realization_index = "1" ;
                :driving_initialization_index = "1" ;
                :driving_physics_index = "2" ;
                :driving_source_id = "CanESM5" ;
                :driving_source_type = "AOGCM" ;
                :target_institution = "Canadian Forest Service, Natural Resources Canada" ;
                :target_institute_id = "CFS-NRCan" ;
                :target_dataset = "ANUSPLIN interpolated Canada daily 300 arc second climate grids" ;
                :target_dataset_id = "ANUSPLIN300" ;
                :target_references = "McKenney, D.W., Hutchinson, M.F., Papadopol, P., Lawrence, K., Pedlar, J.,\n",
                        "Campbell, K., Milewska, E., Hopkinson, R., Price, D., and Owen, T.,\n",
                        "2011. Customized spatial climate models for North America.\n",
                        "Bulletin of the American Meteorological Society, 92(12): 1611-1622.\n",
                        "\n",
                        "Hopkinson, R.F., McKenney, D.W., Milewska, E.J., Hutchinson, M.F.,\n",
                        "Papadopol, P., Vincent, L.A., 2011. Impact of aligning climatological day\n",
                        "on gridding daily maximum-minimum temperature and precipitation over Canada.\n",
                        "Journal of Applied Meteorology and Climatology 50: 1654-1665." ;
                :target_version = "obtained: 2 April 2012, 14 June 2012, and 30 January 2013" ;
                :target_contact = "Pia Papadopol (pia.papadopol@nrcan-rncan.gc.ca)" ;
                :title = "Bias Correction/Constructed Analogue Quantile Mapping version 2.0 (BCCAQ2) downscaling model output for Canada" ;
}

Also new updates of my issues:

• For pr, I added cube.units = 'kg m**-2 s**-1' in the fix file. And esmvaltool got to run w/o reporting further errors, however, I did wonder if this is a proper way to do.
• For the missing scalar coord info for tasmin, I found a similar solution in the documentation of source code for esmvalcore.cmor._fixes.shared. And the issue was resolved!

[zklaus]

Nice progress. I see one problem in the information that you posted, which is the standard name. I refer you to Section 3.3 of the CF conventions. The standard_name must come from the standard_name table. For tasmin, the appropriate standard_name is air_temperature.

[stacyqxz]

Are you suggesting renaming the standard_name? How will this affect the use of esmvaltool? And how am I supposed to rename it?

[zklaus]

The file itself is broken in that it is not a valid CF file. Ideally you can fix your upstream workflow to produce a correct file, i.e. one that reads

...
                tasmin:standard_name = "air_temperature" ;
...

[stacyqxz]

Update on the standard_name:

So I checked the main_log_debug.file, it seems like the wrong standard_name has been auto-fixed to right ones, i.e., air_temperature for tasmin, and precipitation_flux for pr. What does it say to the incompliant standard_name in the original dataset? Cuz I don't think I am able to fix that in the upstream workflow.

[stacyqxz]

So take the file 'tasmin_monthly_average_BCCAQ2v2+ANUSPLIN300_CanESM5_historical+ssp585_r1i1p2f1_1950-2100' as an example:

In the developer_config.yml file, the input_file is defined as {short_name}_monthly_*_BCCAQ2v2+ANUSPLIN300_{dataset}_historical+{exp}_{ensemble}_*-2100.nc.

In the recipe, the dataset is found by key-value pairs {dataset: CanESM5, project: native6, exp: ssp585, ensemble: r1i1p2f1, grid: gn}. short_name is specified under variables in the diagnostic.