Hydra + optuna checkpoints

[konstantinos-p]

Hello!

Is there an option to save the intermediate optimization results of optuna after every hydra job when using hydra --multirun (the best hyperparameter values thus far)? For the moment I am sweeping for a total of 100 jobs using multirun, one crashes and no checkpoint regarding which job was best up to that point is saved. I searched a bit but couldn't find if it was possible. For the moment, all my hydra jobs are part of the same slurm job and the slurm job itself crashes (possibly due to memory leak) so hydra isn't able to catch the error.

[omry]

I feel such logic can belong in the specific sweeper (e.g. OptunaSweeper here).
If you want to implement something like this, we will consider your diff for inclusion.
Regarding Hydra not catching the error of a crash, I am not sure what this means but I suspect it does catch it but simply ignores it.