Parallel Multirun Suggestions

[claysmyth]

Hi,

I'm really enticed by the multirun option in Hydra. However, does anyone have a solid solution on parallelizing the multirun across multiple jobs?

I guess I could always create a wrapper script that uses Ray, and calls a terminal command python script leveraging Hydra with the appropriate parameters for each Ray parallel job... but wondering if anyone has any elegant solution!

Thanks,
Clay

[odelalleau]

Not sure I understand what you want to achieve, could you please clarify?

[claysmyth]

Sure! From Hydra documentation here, a multirun may look like:

python my_app.py --multirun db=mysql,postgresql schema=warehouse,support,school

and my understanding is that each combination of parameters (e.g. mysql and support) would be run sequentially. I am curious if there's a simple solution to run each of these parameter combinations in parallel? This would correspond to all 6 combinations run simultaneously.

A potential solution I see would be to just use Ray and avoid the Hydra multirun altogether... e.g.

dbs = ['mysql', 'postgresql']
schemas=['warehouse', 'support', 'school']

@ray.remote
def call_hydra_script(db, schema):
     os.system(f'python my_app.py db={db} schema={schema}')

runs = [call_hydra_script.remote(combo[0], combo[1]) for combo in itertools.product(dbs, schemas)]
ray.get(runs)

Just curious if a simpler solution is out there! Thanks!

[odelalleau]

and my understanding is that each combination of parameters (e.g. mysql and support) would be run sequentially

This depends on your launcher. For instance with submitit_slurm you can launch all combinations in parallel on a SLURM cluster. You can play around with different launchers to see what works best for you (there's a Ray launcher too but I've never tried it myself: https://hydra.cc/docs/plugins/ray_launcher/)

[claysmyth]

Thanks for that, I'll check it out!