Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarify in docs how runtime parameter resolution works #4096

Merged
merged 7 commits into from
Aug 21, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions RELEASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
## Bug fixes and other changes
* Moved `_find_run_command()` and `_find_run_command_in_plugins()` from `__main__.py` in the project template to the framework itself.
* Fixed a bug where `%load_node` breaks with multi-lines import statements.
* Fixed runtime parameters resolution in `OmegaConfigLoader` by only loading them once.

Check warning on line 9 in RELEASE.md

View workflow job for this annotation

GitHub Actions / vale

[vale] RELEASE.md#L9

[Kedro.weaselwords] 'only' is a weasel word!
Raw output
{"message": "[Kedro.weaselwords] 'only' is a weasel word!", "location": {"path": "RELEASE.md", "range": {"start": {"line": 9, "column": 65}}}, "severity": "WARNING"}

Check warning on line 9 in RELEASE.md

View workflow job for this annotation

GitHub Actions / vale

[vale] RELEASE.md#L9

[Kedro.words] Use 'after' instead of 'once'.
Raw output
{"message": "[Kedro.words] Use 'after' instead of 'once'.", "location": {"path": "RELEASE.md", "range": {"start": {"line": 9, "column": 83}}}, "severity": "WARNING"}

## Breaking changes to the API

Expand Down
4 changes: 3 additions & 1 deletion kedro/config/omegaconf_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -343,7 +343,9 @@ def load_and_merge_dir_config( # noqa: PLR0913
if not aggregate_config:
return {}

if key == "parameters":
# Only merge in runtime parameters once for the base env
is_base_env = conf_path.endswith(self.base_env)
if key == "parameters" and is_base_env:
# Merge with runtime parameters only for "parameters"
return OmegaConf.to_container(
OmegaConf.merge(*aggregate_config, self.runtime_params), resolve=True
Expand Down
39 changes: 39 additions & 0 deletions tests/config/test_omegaconf_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -1068,6 +1068,45 @@ def test_runtime_params_resolution(self, tmp_path):
# runtime params are resolved correctly in catalog
assert conf["catalog"]["companies"]["type"] == runtime_params["dataset"]["type"]

def test_runtime_params_resolution_with_env(self, tmp_path):
base_params = tmp_path / _BASE_ENV / "parameters.yml"
prod_params = tmp_path / "prod" / "parameters.yml"
runtime_params = {
"aaa": {
"bbb": {
"abb": "2011-11-11",
}
}
}
param_config = {
"aaa": {
"bbb": {
"aba": "2023-11-01",
"abb": "2023-11-01",
"abc": 14,
}
},
"xyz": {"asdf": 123123},
}
prod_param_config = {"def": {"gg": 123}}
_write_yaml(base_params, param_config)
_write_yaml(prod_params, prod_param_config)
conf = OmegaConfigLoader(
tmp_path,
base_env=_BASE_ENV,
default_run_env="prod",
runtime_params=runtime_params,
)

expected_parameters = {
"aaa": {"bbb": {"aba": "2023-11-01", "abb": "2011-11-11", "abc": 14}},
"xyz": {"asdf": 123123},
"def": {"gg": 123},
}

# runtime parameters are resolved correctly across parameter files from different environments
assert conf["parameters"] == expected_parameters

def test_runtime_params_missing_default(self, tmp_path):
base_params = tmp_path / _BASE_ENV / "parameters.yml"
runtime_params = {
Expand Down