recipe_ocean_amoc.yml with CMIP6 data in JASMIN

[RChapman1407]

Hello,

I am trying to run the example recipe_ocean_amoc.yml, but now for certain CMIP6 data. I can run it in JASMIN fine for CMIP5, but keep getting errors when I change to CMIP6. Specifically I would like to run it for picontrols of CanESM5, MPI-ESM1-2-LR, and HadGEM3. I only need to calculate the amoc at 26N, I do not need any of the other outputs that this script makes.

The changes I have made are: exp:piControl, and I have added grid:gn and mip:Omon.
Should I be adding further descriptors for the script to be able to find the necessary data?

The error I get looks like this;

2023-03-20 17:13:05,886 UTC [49107] INFO    Skipping: no data found for {'short_name': 'msftmz', 'optional': True, 'mip': 'Omon', 'derive': True, 'force_derivation': False, 'variable_group': 'amoc', 'diagnostic': 'diag_timeseries_amoc', 'preprocessor': 'default', 'dataset': 'CanESM5', 'project': 'CMIP6', 'exp': 'piControl', 'ensemble': 'r1i1p1f1', 'grid': 'gn', 'recipe_dataset_index': 0, 'timerange': '1900/1910', 'institute': ['CCCma'], 'activity': 'CMIP', 'alias': 'CanESM5', 'original_short_name': 'msftmz', 'standard_name': 'ocean_meridional_overturning_mass_streamfunction', 'long_name': 'Ocean Meridional Overturning Mass Streamfunction', 'units': 'kg s-1', 'modeling_realm': ['ocean'], 'frequency': 'mon', 'start_year': 1900, 'end_year': 1910} which is marked as 'optional'
2023-03-20 17:13:05,887 UTC [49107] INFO    Skipping: no data found for {'short_name': 'msftyz', 'optional': True, 'mip': 'Omon', 'derive': True, 'force_derivation': False, 'variable_group': 'amoc', 'diagnostic': 'diag_timeseries_amoc', 'preprocessor': 'default', 'dataset': 'CanESM5', 'project': 'CMIP6', 'exp': 'piControl', 'ensemble': 'r1i1p1f1', 'grid': 'gn', 'recipe_dataset_index': 0, 'timerange': '1900/1910', 'institute': ['CCCma'], 'activity': 'CMIP', 'alias': 'CanESM5', 'original_short_name': 'msftyz', 'standard_name': 'ocean_y_overturning_mass_streamfunction', 'long_name': 'Ocean Y Overturning Mass Streamfunction', 'units': 'kg s-1', 'modeling_realm': ['ocean'], 'frequency': 'mon', 'start_year': 1900, 'end_year': 1910} which is marked as 'optional'
2023-03-20 17:13:05,887 UTC [49107] ERROR   Could not create all tasks
2023-03-20 17:13:05,887 UTC [49107] ERROR   Invalid value encountered for `timerange`. Valid values must be separated 
by `/`. Got [''] instead.
2023-03-20 17:13:05,979 UTC [49107] INFO    Maximum memory used (estimate): 0.2 GB
2023-03-20 17:13:05,980 UTC [49107] INFO    Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
2023-03-20 17:13:05,981 UTC [49107] ERROR   Could not create all tasks

[bouweandela]

Could you please post the exact recipe that you're trying to run? You can rename it so it ends with .txt if GitHub doesn't allow uploading it.

[RChapman1407]

Here is the full script I am trying to use, here just for CanESM5:

# ESMValTool
# recipe_ocean_amoc.yml
---
documentation:
  title: Atlantic Meridional Overturning Circulation Recipe

  description: |
    Recipe to produce time series figures of the derived variable, the
    Atlantic meriodinal overturning circulation (AMOC).
    This recipe also produces transect figures of the stream functions for
    the years 2001-2004.

  authors:
    - demora_lee

  maintainer:
    - demora_lee

  references:
    - demora2018gmd

  projects:
    - ukesm

diagnostics:
  # --------------------------------------------------
  # Time series diagnostics
  # --------------------------------------------------
  diag_timeseries_amoc:
    description: atlantic_meridional_overturning_circulation
    variables:
      amoc:
        mip: Omon
        derive: true
       force_derivation: false
    additional_datasets:
 - {dataset: CanESM5,            project: CMIP6,   mip: Omon,  exp: piControl,   ensemble: r1i1p1f1, grid: gn,  start_year: 1900,  end_year: 1910}
 scripts:
      AMOC_timeseries:
        script: ocean/diagnostic_timeseries.py
        moving_average: 1 years

Thanks, any help at all will be apreciated!

[ehogan]

Hi @RChapman1407 👋 Thank you for getting in touch! 😊 I wanted to let you know that I edited your post to add triple back ticks (```) around the recipe text so that it renders as code (which makes it easier to read!) 🥳

I believe the error you are getting is due to the fact that the msftmz variable isn't available on JASMIN (in /badc/cmip6/data/CMIP6/CMIP/CCCma/CanESM5/piControl/r1i1p1f1/Omon/).

A slightly longer explaination: if I'm following the code correctly, the derivation of the amoc variable requires either msftmz and msftyz for CMIP6 (see preprocessor/_derive/amoc.py). However, according to the CMIP6-CEDA Data Search, the msftyz variable is not available for the facets detailed in your dataset. So the msftmz variable would be used, if the data were available on JASMIN.

I downloaded the msftmz variable locally and your recipe ran successfully using ESMValTool v2.8.0 (although I had to remove the start_year and end_year as the available years for this data appear to start in 5201 and end in 6200).

It should be possible to download the msftmz variable yourself, see ESMValTool Documentation: Obtaining input data: Models. Alternatively, the data could be made available on JASMIN, although I'm not sure how to go about this; would someone from the @ESMValGroup/technical-lead-development-team be able to help here, please? 😊

I hope that helps? Please let me know if you have any other questions! 😁

[bouweandela]

Thanks for looking into this @ehogan! CMIP6 data can be downloaded automatically by the tool, provided that offline: false in config-user.yml or if you're using the latest version (v2.8) of the tool, search_esgf: when_missing or search_esgf: always in config-user.yml.

Did the advice by @ehogan solve your issue @RChapman1407?

[RChapman1407]

Thank you, the issue makes sense now. I haven't manage to fix it quite yet, but this certainly puts me along the right lines