Question about parallel sweeping - idle workers

[lifengjin]

I am trying to use the Optuna sweeper for hyperparameter tuning with various configurations. One thing I was not able to figure out is how to make workers not idle. I have set n_jobs to 8 and used Joblib as the launcher, and can confirm that initially 8 parallel jobs were launched with one GPU assigned to one job. But when some jobs finished, the GPUs became idle because nothing was launched. Because some of the jobs were longer than others, I would prefer workers to run other jobs instead of being idle. Is there a way to do this？

[Jasha10]

Currently there is no way to do this, as batch/job scheduling of the optuna sweeper is based on a synchronous model where all 8 jobs in the batch must finish before the next batch is begun. The Optuna sweeper's scheduling model is an area where there's plenty of room for improvement.
A related issue is #692.

[lifengjin]

Thanks for the answer. The next question is if any of the sweepers can do this?

[Jasha10]

No; all of Hydra's official sweepers are batch-based and have the same behavior (where all jobs from the current batch must finish before the next batch begins).