Also read default custom CMOR tables if custom location is specified

[schlunma]

Description

This PR makes sure that the default custom CMOR tables are always read, even when a custom location is specified.

Closes #2278

Link to documentation: https://esmvaltool--2279.org.readthedocs.build/projects/ESMValCore/en/2279/quickstart/configure.html#custom-cmor-tables

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Unit tests have been added
• 🛠 Changes are backward compatible
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful (Codacy issues were already present in main).

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (f515169) 93.61% compared to head (073c38a) 93.61%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2279   +/-   ##
=======================================
  Coverage   93.61%   93.61%           
=======================================
  Files         238      238           
  Lines       13113    13113           
=======================================
  Hits        12276    12276           
  Misses        837      837           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[schlunma]

@FranziskaWinterstein could you try if this solves your problem with the custom tables? Thanks!!

[valeriupredoi]

many thanks, Manu! A couple very minor points from me 🍺

[valeriupredoi]

BTW @schlunma I think this closes this issue too #1388 🍺

[schlunma]

BTW @schlunma I think this closes this issue too #1388 🍺

Nah I don't think so, this PR only changes the behavior if a custom location for the custom location is specified, #1388 also appears with our default location for the custom tables (I think 😁).

[valeriupredoi]

ah well, worth the try, am in Xmas cleaning mode 😁

[FranziskaWinterstein]

@schlunma Thank you for taking care of this. However, I have difficulties in executing my recipe. I was able to reproduce the former error, but with your changes the following happens:

Traceback (most recent call last):
  File "/home/b/b309109/ESMValCore/esmvalcore/_main.py", line 518, in run
    fire.Fire(ESMValTool())
  File "/home/b/b309109/ESMValCore/esmvalcore/_main.py", line 319, in __init__
    self.__setattr__(entry_point.name, entry_point.load()())
  File "/work/bd1132/b309109/conda_envs/envs/esmvaltool/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/work/bd1132/b309109/conda_envs/envs/esmvaltool/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/b/b309109/ESMValTool/esmvaltool/cmorizers/data/cmorizer.py", line 17, in <module>
    from esmvalcore._task import write_ncl_settings
  File "/home/b/b309109/ESMValCore/esmvalcore/_task.py", line 24, in <module>
    from ._citation import _write_citation_files
  File "/home/b/b309109/ESMValCore/esmvalcore/_citation.py", line 10, in <module>
    from .config._diagnostics import DIAGNOSTICS
  File "/home/b/b309109/ESMValCore/esmvalcore/config/__init__.py", line 11, in <module>
    from ._config_object import CFG, Config, Session
  File "/home/b/b309109/ESMValCore/esmvalcore/config/_config_object.py", line 242, in <module>
    CFG = Config._load_user_config(USER_CONFIG, raise_exception=False)
  File "/home/b/b309109/ESMValCore/esmvalcore/config/_config_object.py", line 70, in _load_user_config
    new.update(mapping)
  File "/work/bd1132/b309109/conda_envs/envs/esmvaltool/lib/python3.10/_collections_abc.py", line 999, in update
    self[key] = other[key]
  File "/home/b/b309109/ESMValCore/esmvalcore/config/_validated_config.py", line 65, in __setitem__
    raise InvalidConfigParameter(
esmvalcore.exceptions.InvalidConfigParameter: `offline` is not a valid config parameter.

[schlunma]

It looks like you are using the removed option offline in one of your config files (could also be the default one at ~/.esmvaltool/config-user.yml). offline has been deprecated in v2.8 and removed just now for v2.10 (#2213). The new option is called search_esgf (details here).

[schlunma]

See #2280

[FranziskaWinterstein]

Okay, after solving all my config issues, the behaviour concerning the custom tables is as I would expect it, i.e. favouring the external folder over the internal only if duplicate tables are defined. Thank you!

[valeriupredoi]

@schlunma - I approved this PR last year 😁 I can also merge if I get the nod of OK from you 👍

[schlunma]

@schlunma - I approved this PR last year 😁 I can also merge if I get the nod of OK from you 👍

Yeah, please merge 😁