Optimizing efficiency using deepspeed

[base-y]

Hi, I am using ZeRO-3 for training my T5 base model. But I see samplespersec=24 which I think is inefficient. What are the things I can try to improve the throughput? Also, can I know the maximum possible throughput I can get (maybe by doing sample runs)? I am relatively new to deepspeed.

Also, sometimes, throughput w/o using deepspeed seem to be higher than w/ deepspeed (using 8 GPUs to train). Would like to know the scenarios when deepspeed doesnt help much.

[tjruwase]

The ZeRO optimizations in DeepSpeed are most helpful when:

1. Model is too large to train using data parallelism alone.
2. Larger batch sizes can improve compute efficiency without hurting model performance.

The memory saving of ZeRO is a trade-off for increased communication. The memory saving and communication overhead increase with ZeRO stages. The communication overhead can hurt throughput of smaller models like t5-base, which don't benefit much from the memory savings. In such cases, it is probably better to disable ZeRO by setting the stage to 0. You might find the Flops Profiler or Autotuner helpful for your investigation.