Support the Whisper model in onnxruntime-genai

[baijumeswani]

Many changes in this PR are from @RyanUnderhill's whisper branch.

This pull-request introduces support for running the openai/whisper model. In particular, this pull-request introduces the following:

• Whisper preprocessing: Given an audio file, the preprocessing stage creates the mel spectrogram and the decoder input ids that are needed for the model execution.

• Whisper model execution: Given the log mel spectrogram and the decoder input ids, the whisper onnx models can be executed. The model execution is split into three phases

  • EncoderDecoderInit: The corresponding ONNX model for this phase contains the Encoder and the Decoder with the Attention operator. This model is run on the first token generation only.
  • DecoderInit: This phase manages the transition from first token generation to the second token generation. i.e. From managing the inputs/outputs of the EncoderDecoderInit ONNX models to managing the inputs/outputs of the Decoder ONNX model.
  • Decoder: This phase manages all the remaining token generation steps. The corresponding ONNX model contains only the decoder logic with the DecoderMaskedMultiHeadAttention operator.

  The model execution also manages the outputs of the model: logits and optionally the cross_qk buffers required for computing the word level timestamps.

• The Python, C, C++ APIs for loading audios, preprocessing the audio files, and executing the whisper model.

With this pull-request, onnxruntime-genai can execute the openai/whisper model on CPU (fp32) and CUDA (fp16 and fp32) EPs with batch_size >= 1 and beam_size >= 1

Changes required that are not part of this pull-request:

• Making the cross_qk buffers available as outputs for the CPU EP.
• Refining the user API.
• Add C# API to load multiple audio files for batch size > 1.
• Add C# example.
• Splitting the phases into two Encoder and Decoder to avoid the decoder weight duplication in EncoderDecoderInit.
• An end-to-end example with word-level timestamps

[yufenglee]

we need to add some tests in following PR.

[yufenglee]

and add a description on how to create the model

---

In reply to: 2347714980