How do I use build_experiment with a HPC?

[NeilGirdhar]

The Running on HPC section only shows the orion hunt command, but the Parallel Workers section says that using the Python API is possible. How do I do this? Does running experiment.workon on a HPC automatically spawn jobs?

[bouthilx]

Hi! You can use experiment.workon with the Dask backend if you setup a Dask SlurmCluster (https://jobqueue.dask.org/en/latest/). This can be a bit tricky since it requires a long running job for the SlurmCluster to run, from which it will spawn the other jobs.

The simplest solution is to scale your jobs the same way you would do with orion hunt but instead calling your script. If you follow the tutorial on the python api (https://orion.readthedocs.io/en/stable/auto_tutorials/code_1_python_api.html), you can scale this by simply running the same script in different jobs. Each execution gets synchronized with the others through the database. As long as they point to the same experiment name and version, these will be working on the same HPO.

[NeilGirdhar]

Thanks! I think I'm going to implement this in two ways:

• A multi-job approach that does not use orion hunt, but rather uses build_experiment and report_results, and
• A multi-task approach that uses build_experiment and workon.

The problem with the orion hunt approach is that it puts the hyperparameter specification into the shell script, which I don't want.

[bouthilx]

Indeed, you can parallelize in both ways at the same time! Much like running multiple times orion hunt --n-workers 10.

Just in case you didn't know, you could use config files if you don't want the hyperparameters in the commandline itself: https://orion.readthedocs.io/en/stable/user/searchspace.html#configuration-file.

[NeilGirdhar]

For anyone looking for an answer to this, here's my code:

def experiment_f(demo: Demo,
                 log: LoggingManager,
                 progress_manager: ProgressManager,
                 **params: Any) -> list[dict[str, Any]]:
    solvers = demo.create_solvers(**params)
    demo_loss = demo.demo_loss(solvers, log, progress_manager)
    for solver in solvers:
        solver.delete_cache()
    return [{'name': 'demo_loss',
             'type': 'objective',
             'value': demo_loss}]

def run_demo(mode: VisualizationMode,  # noqa: C901
             demo: Demo, *,
             workers: int,
             trials: int,
             failures: int,
             enable_log: bool
             ) -> None:
    experiment = build_experiment(name=demo.name, space=demo.metaparameter_space(),
                                  max_trials=trials, max_broken=failures,
                                  algorithm={"tpe": {"n_initial_points": 5}})
    if mode == VisualizationMode.multi_node:
        with solver_context_manager(thread_limit=None):
            if workers == 0:
                workers = 1
            assert workers > 0
            for _ in range(workers):
                trial = experiment.suggest()
                params = trial.params
                console_log.info_generic(params)
                results = experiment_f(demo, log=console_log,
                                       progress_manager=null_progress_manager, **params)
                experiment.observe(trial, results)
            return
    with solver_context_manager(thread_limit=1):
        assert mode == VisualizationMode.multi_task
        assert workers > 0
        with experiment.tmp_executor("joblib", n_workers=workers):
            experiment.workon(experiment_f, workers, demo=demo, log=console_log,
                              progress_manager=null_progress_manager)
        experiment.plot.regret().show()