When is output_subdir created || override output_subdir and hydra.run.dir after instantiation

[JuanFMontesinos]

Hello there,
I'm trying to nicely integrate hydra with MLFlow. I've started with the a mnist toy example :)
First of all I'm pretty begginer with hydra.

@hydra.main(config_path="../conf", config_name="PLMnist")
def train(cfg: src.train.hydra.PLMnist) -> None:
    with mlflow.start_run(run_name=cfg.mlflow.run_name) as run:

Hydra creates the output folder at hydra.run.dir and then copies the config at output_subdir.
Then I instantiate the mlflow experiment and it creates its own file structure which is
mlruns->experiment->run

Actually, I would like to override hydra.run.dir to point towards mlruns.experiment.run
The main clash is that cofig files are read by hydra and then the MLflow experiment is created, but I cannot know before hand the mlflow experiment path.

Is this possible without reinstantiating hydra and without breakiung the internal way of working?

My attempt has been instantiating hydra to obtain the cfg, then just clearing the instance and calling it again with the mlflow path
So a pseudo code would be

@hydra.main(config_path="../conf", config_name="PLMnist")
def train(cfg: src.train.hydra.PLMnist) -> None:
    with mlflow.start_run(run_name=cfg.mlflow.run_name) as run:
        from hydra import initialize, initialize_config_module, initialize_config_dir
        from hydra.core.global_hydra import GlobalHydra
        from hydra import compose, initialize_config_dir

        GlobalHydra.instance().clear()
        initialize_config_dir(config_dir=str(cfg_path))
        cfg = compose(
            config_name=cfg_name, overrides=[f"hydra.run.dir='{mlflow_exp_dir}'"]
        )
        

I've notice that this way the .hydra folder (output_subdir) is not created at hydra.run.dir/output_subdir (namely mlflow_exp_dir/.hydra)
So I assume this is never done in the compose function.

In summary the questions would be:

1. How can I force hydra to create the output_subdir folder?
2. Is it better re-instantiating hydra or simply overriding the hydra.run.dir after my mlflow experiment is created?

[odelalleau]

What if you create a symlink in your mlflow folder that points to Hydra's run dir? This way you can access everything from your mlflow folder without messing with any of Hydra's internals.

[JuanFMontesinos]

Hi @odelalleau thanks for the suggestion.
That's actually a good option. I could manage to do the following although it's a bit ugly:

def main(*fn_args, **kwargs):
    global experiment
    global run
    exp_name = None
    run_name = None

    from hydra import main as _main

    args = sys.argv[1:]
    for arg in args:
        if "mlflow.exp_name" in arg:
            exp_name = arg.split("=")[1]
        elif "mlflow.run_name" in arg:
            run_name = arg.split("=")[1]
    experiment = mlflow.set_experiment(experiment_name=exp_name)
    run = mlflow.start_run(run_name=run_name)
    experiment_dir = Path(re.sub(r"file://", "", run.info.artifact_uri))
    assert experiment_dir.exists(), f"{experiment_dir} does not exist"
    experiment_dir = experiment_dir.parent
    sys.argv.append(f"++hydra.run.dir={experiment_dir.as_posix()}")

    return _main(*fn_args, **kwargs)

Esentially creates a main function that mimics hydra's one without touching the internals. Just injecting the path before hydra.main is called. Do you find anything where this solution can break hydra internally?

[odelalleau]

Hydra should work fine, but this implementation has some limitations:

1. It forces you to specify the mlflow settings as command line arguments (you can't have them in the config itself)
2. The run object isn't available anymore (I don't know if you might need it elsewhere later)

If my symlink idea doesn't work for you, you can also try moving files from hydra's run dir to your mlfflow folder. Linux usually plays rather nicely with opened files being moved (like logs). Can't guarantee this will work 100% though (you might need to create symlinks inside hydra's run dir to point to the moved files).