What is the best Practice for Structured Configurations with Shared Callbacks in Nested Structure

[icedpanda]

I am trying to set up a configuration structure using Hydra that has shared callbacks for different tasks, while still maintaining a nested structure for each task. The goal is to reduce redundancy in the configuration while preserving the hierarchical structure (pretrain and finetuning in this case).

Given the structure:

• pretrain and finetuning are two tasks
• Both tasks share common callbacks configuration like model_summary, rich_progress_bar, etc.
• There are also task-specific override for early_stopping and model_checkpoint
• Due to the complexity of the training process, I'm using multiple trainer.fit() calls. For the sake of this issue, let's assume we have two stages: pre-training and fine-tuning.

However, I've been unable to combine the usage of defaults with self to achieve my desired structure. The configurations end up being outside the pretrain and finetuning namespaces. From this tutorial, I can manage to have the nest callbacks without using defaults and _self_ like:

callbacks:
   pretrain:
       # some callbacks here
   fine-tuning:
      # some callbacks here

Ideally, I would like to have

# callbacks/common.yaml
defaults:
  - model_checkpoint
  - early_stopping
  - model_summary
  - rich_progress_bar
  - _self_

model_checkpoint:
  dirpath: ${paths.output_dir}/checkpoints
  monitor: "val/epoch_loss"
  save_last: False
  auto_insert_metric_name: False

model_summary:
  max_depth: -1

lr_monitor:
  _target_: pytorch_lightning.callbacks.LearningRateMonitor
  logging_interval: step

# callbacks/PF
defaults:
  - tasks/pretrain
  - tasks/fine-tuning

#callbacks/tasks/pretrain
defaults:
 - common
 - _self_

model_checkpoint:
  dirpath: ${paths.output_dir}/checkpoints
  monitor: "val/pretrain/epoch_loss"
  mode: "min"
  save_last: False
  auto_insert_metric_name: False

#callbacks/tasks/fine_tuning
defaults:
 - common
 - _self_

model_checkpoint:
  dirpath: ${paths.output_dir}/checkpoints
  monitor: "val/fine_tuning/acc"
  mode: "max"
  save_last: False
  auto_insert_metric_name: False

Given the above, how can I best structure my configurations using Hydra to achieve the desired nested structure while utilizing the power of defaults and _self_? Is there a recommended way or best practice for such use cases?

[odelalleau]

Sorry, there's a lot to digest here and I'm not sure I fully understand what you're trying to achieve... but I wonder if what you might be looking for is something like

defaults:
- /callbacks@callbacks.pretrain: model_checkpoint
- /callbacks@callbacks.fine-tuning: early_stopping
...

where you would have callbacks/model_checkpoint.yaml, callbacks/early_stopping.yaml, ...

[icedpanda]

Thanks @odelalleau, works by change it to:

defaults:
  - /callbacks@tasks.pretrain: tasks/pretrain
  - /callbacks@tasks.fine_tuning: tasks/fine_tuning