Can I access all config groups during runtime?

[hogru]

Hi, sorry if this is a basic question, I am hydra noob. I have several config groups with the corresponding defaults in config.yaml. So far, so good.

Is there a way that I can access the names of all potential config group options aside from the default config during runtime?

So, following the examples in the documentation, I would have a db config group and mysql is the default. During runtime, can I get a list of all config group options, i.e., mysql and postgresql?

In my case, this is a config group of encodings, of which I use one (the default), but I want to do a bit with the other config group options.

[Jasha10]

Hi @hogru,

Currently Hydra does not have an API for getting a list of available config group options at runtime.
The closest thing I can think of is Hydra's CLI completion support, which can list available options at the command line.

You could reverse-engineer this CLI completion if you don't mind a text-based solution. Using my_app.py and conf from this example app, I got the list of completions "db=mysql db=postgresql" as follows using bash:

$ COMP_LINE='python my_app.py db=' python my_app.py -sc query=bash
db=mysql db=postgresql

You could do this via a python script using a subprocess:

# get_completions.py
import subprocess
import os

stdout = subprocess.run(
    ["python", "my_app.py", "-sc", "query=bash"],
    env={"COMP_LINE": "python my_app.py db=", **os.environ},
    capture_output=True
).stdout

print(stdout.decode())

$ python get_completions.py
db=mysql db=postgresql

Anyway, feel free to open a feature request for a less-hacky way to get a list of config group options :)

[hogru]

Thank you @Jasha10 for taking the time for this elaborate answer. For now, it's probably easier to manually keep the config group options and a global list in sync, but I learned something from your approach which might be helpful in a completely other corner of the maze.

[jieru-hu]

if you use ConfigStore, you can probably look into using the list call and see if it fits your need.

[hogru]

Thank you @jieru-hu. I experimented a bit but without success. I can run

cs = ConfigStore.instance()
cs.store(name="cfg", node=MolGenConfig)
print(cs.list("hydra"))

and the result is ['config.yaml']. But I could not pass any other variable to cs.list() without errors. Looking with the debugger into csI find the structure of my config, but they don't have values (well they have: ??? or MISSING) so I would not understand where list() gets the values from. Once I am in the decorated main() function, I have a config object with all config values set (but no config groups).

If you feel I am close and need just another hint, I am happy to get a push in the right direction, and otherwise I will just "copy" the config group options into a global variable.