Fix PyTorch CI HUD dashboard missing perf numbers: hf_Whisper

[xmfan]

A few models were passing accuracy check, but surprisingly failing the perf run, resulting in dashboard entries like:

Reproing the hud's commands locally,

# pass
python benchmarks/dynamo/torchbench.py --accuracy --no-translation-validation --training --amp --backend inductor --disable-cudagraphs --device cuda --total-partitions 4 --partition-id 1 --output hf_Whisper_accuracy.csv --only hf_Whisper

# fail (on https://github.com/pytorch/benchmark/blob/4ea3bba3b8010f5d4a629bb8f530a92570f34518/torchbenchmark/util/model.py#L195C48-L195C48)
python benchmarks/dynamo/torchbench.py --performance --cold-start-latency --training --amp --backend inductor --disable-cudagraphs --device cuda --total-partitions 4 --partition-id 1 --output hf_Whisper_perf.csv --only hf_Whisper

The error suggests that hf_Whisper does not provide a batch size for the training mode perf run.

Summarizing discussion with @xuzhao9:

I think we could:

1. set a default train batch size for hf_Whisper, if you still want to test forward/backward pass without a defined train test
2. in model.py, make sure self.batch_size is not None (before accuracy check overrides batch size to 4)

I implement 1, we set default batch sizes in the parent class of all benchmark models, with ability to be overwritten by individual models.

[xuzhao9]

The code is much clearer now. Thanks for making this improvement!

[facebook-github-bot]

@xmfan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[msaroufim]

A lot of models are missing, I'm curious how many more models are affected by this issue cc @bdhirsh

[facebook-github-bot]

@xmfan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@xmfan merged this pull request in 3f1c3eb.

[eellison]

I don't understand why a model would have ALLOW_CUSTOMIZE_BSIZE but we would end up in this branch. For context, I'm looking into why we are not running stable_diffusion_unet in inference

[eellison]

Why don't we just use the batch size of the model instead of failing ?

[xuzhao9]

If ALLOW_CUSTOMIZE_BSIZE = False, the model will only accept the default batch size, not the batch size specified by the user.

We could silently use the default batch size instead of failing, but my concern this will cause misunderstanding on the user side (for example, they might think the model is running in batch size 100, but ALLOW_CUSTOMIZE_BSIZE = False and the default batch size is 1, so it will run silently with batch size 1)

[eellison]

If batch_size is passed in as None, it seems okay to use the default specified on the model, instead of specified in self.metadata["devices"][current_device_name][device_batch_size_key]

[eellison]

Also, if we're worried about that case, we should also fix this upstream handling of it: https://github.com/pytorch/pytorch/blob/main/benchmarks/dynamo/torchbench.py#L369-L374

[xuzhao9]

If batch_size is passed in as None, it seems okay to use the default specified on the model, instead of specified in self.metadata["devices"][current_device_name][device_batch_size_key]

Right, this is a bug. If ALLOW_CUSTOMIZE_BSIZE = False and batch_size passed in as None, we should use the default specified on the model instead of the device-specified batch size.