Config setup docs - through tests

[stan-dot]

please validate the directional correctness of the test first. #tdd

[stan-dot]

it works until line 179 ( call to loader.use_values_from_yaml)

[stan-dot]

    config_data = {
        "env": {},
        "sources": [
            {"kind": "dodal", "module": "dodal.adsim"},
            {"kind": "planFunctions", "module": "dls_bluesky_core.plans"},
            {"kind": "planFunctions", "module": "dls_bluesky_core.stubs"},
        ],
        "data_writing": {
            "visit_directory": "/dls/p38/data/2023/cm33874-1",
            "group_name": "BL38P",
        },
    }

does not map directly to this:

#532 (comment)

perhaps config loading could be simplified, to mirror more closely the helm structure

[stan-dot]

all good

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.56%. Comparing base (ac0fee8) to head (2483a41).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #634   +/-   ##
=======================================
  Coverage   92.56%   92.56%           
=======================================
  Files          35       35           
  Lines        1654     1655    +1     
=======================================
+ Hits         1531     1532    +1     
  Misses        123      123           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[callumforrester]

Should: This seems like a lot of extra boilerplate compared to just writing the example config below into a YAML file.

[stan-dot]

I'd say it's an opinionated tradeoff (some more lines in a fixture + a dict-like test content) vs a bunch of test-data yaml files .

[callumforrester]

Perhaps, the main motivation is that you may as well have the thing the user will write as an input to your test data

[stan-dot]

class BasicAuthentication(BaseModel):
    """
    User credentials for basic authentication
    """

    username: str = Field(description="Unique identifier for user")
    password: Secret[str] = Field(description="Password to verify user's identity")

    @field_validator("username", "password", mode="before")
    @classmethod
    def get_from_env(cls, v: str):
        if v.startswith("${") and v.endswith("}"):
            return os.environ[v.removeprefix("${").removesuffix("}").upper()]
        return v

current error is coming from bluesky-stomp, where for some reason it uses the class method to get the password from env. therefore full validation from a file might require an adjustment

[stan-dot]

@ZohebShaikh any ideas?

[DiamondJoseph]

class BasicAuthentication(BaseModel):
    """
    User credentials for basic authentication
    """

    username: str = Field(description="Unique identifier for user")
    password: Secret[str] = Field(description="Password to verify user's identity")

    @field_validator("username", "password", mode="before")
    @classmethod
    def get_from_env(cls, v: str):
        if v.startswith("${") and v.endswith("}"):
            return os.environ[v.removeprefix("${").removesuffix("}").upper()]
        return v

current error is coming from bluesky-stomp, where for some reason it uses the class method to get the password from env. therefore full validation from a file might require an adjustment

AttributeError: 'Secret' object has no attribute 'startswith'

@stan-dot you need to unwrap the Secret[str] before using it.

It fetches it from the ENV if configured to do so for injection of the RMQ password. If you want to adjust how the handling of that env var fetching is done to your override levels handling then feel free. @keithralphs needs to mount a bunch of tracing values as env vars, so having that handling in one place is preferable. But the type of the password should remain Secret[str] to prevent accidental logging.

[stan-dot]

File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/bluesky_stomp/models.py", line 17, in get_from_env
if v.startswith("${") and v.endswith("}"):
AttributeError: 'Secret' object has no attribute 'startswith'

odd, digging deeper

[stan-dot]

FAILED tests/unit_tests/test_config.py::test_config_yaml_parsed_complete[temp_yaml_config_file0] - blueapi.utils.invalid_config_error.InvalidConfigError: Something is wrong with the configuration file: 
 {'type': 'model_type', 'loc': ('scratch', 'repositories', 0), 'msg': 'Input should be a valid dictionary or instance of ScratchRepository', 'input': 'dodal', 'ctx': {'class_name': 'ScratchRepository'}, 'url': 'https://errors.pydantic.dev/2.9/v/model_type'}
FAILED tests/unit_tests/test_config.py::test_config_yaml_parsed_complete[temp_yaml_config_file1] - blueapi.utils.invalid_config_error.InvalidConfigError: Something is wrong with the configuration file: 
 {'type': 'model_type', 'loc': ('scratch', 'repositories', 0), 'msg': 'Input should be a valid dictionary or instance of ScratchRepository', 'input': 'dodal', 'ctx': {'class_name': 'ScratchRepository'}, 'url': 'https://errors.pydantic.dev/2.9/v/model_type'}
FAILED tests/unit_tests/worker/test_task_worker.py::test_worker_and_data_events_produce_in_order - concurrent.futures._base.TimeoutError
============= 3 failed, 

got a more specific error