Getting error "Inconsistent number of segments: whisper_segments ({l1}"

[amitli1]

Sometimes I'm getting error:

line 870, in _transcribe_timestamped_efficient
    assert l1 == l2 or l1 == 0, f"Inconsistent number of segments: whisper_segments ({l1}) != timestamped_word_segments ({l2})"
AssertionError: Inconsistent number of segments: whisper_segments (1) != timestamped_word_segments (0)

when using whisper.transcribe.

I'm running with the following code:

whisper.transcribe(
                model,
                audio,             
                initial_prompt=buffer, # compilation of all transcript history
                verbose=True  
            )

What is wrong ?

[Jeronymous]

Thanks for reporting. Too bad to have this error again.

I would need more details to be able to reproduce and investigate:
some audio (it can be a pickled torch array, if not a file), the details of the model that is used, as well as the initial_prompt.

[dominikmau]

@Jeronymous
Hello, thanks for the great project.
Unfortunately I get the same error:

Inconsistent number of segments: whisper_segments (1) != timestamped_word_segments (2)
Traceback (most recent call last):
  File "C:\Users\test\Desktop\whisper2\whisper_test.py", line 7, in <module>
    result = whisper.transcribe(model, audio, language="de")
  File "C:\Users\test\Desktop\whisper2\venv\lib\site-packages\whisper_timestamped\transcribe.py", line 296, in transcribe_timestamped
    (transcription, words) = _transcribe_timestamped_efficient(model, audio,
  File "C:\Users\test\Desktop\whisper2\venv\lib\site-packages\whisper_timestamped\transcribe.py", line 920, in _transcribe_timestamped_efficient
    assert l1 == l2 or l1 == 0, f"Inconsistent number of segments: whisper_segments ({l1}) != timestamped_word_segments ({l2})"
AssertionError: Inconsistent number of segments: whisper_segments (1) != timestamped_word_segments (2))

Happens to me with every audio file, here is an example - test.mp3, 00:00:56:
https://mega.nz/file/EWpCiD5b#FqW20Gy-StHcd9dskoOe3pEPuFxSJyzfOUhk6kd4NmQ

I use a finetuned model:
https://huggingface.co/primeline/whisper-large-v3-german

Python version:

Python 3.10.11

Installed Python packages:

certifi==2024.7.4
charset-normalizer==3.3.2
colorama==0.4.6
Cython==3.0.10
dtw-python==1.5.1
filelock==3.13.1
fsspec==2024.2.0
huggingface-hub==0.23.4
idna==3.7
intel-openmp==2021.4.0
Jinja2==3.1.3
llvmlite==0.43.0
MarkupSafe==2.1.5
mkl==2021.4.0
more-itertools==10.3.0
mpmath==1.3.0
networkx==3.2.1
numba==0.60.0
numpy==1.26.3
openai-whisper==20231117
packaging==24.1
pillow==10.2.0
PyYAML==6.0.1
regex==2024.5.15
requests==2.32.3
safetensors==0.4.3
scipy==1.14.0
sympy==1.12
tbb==2021.11.0
tiktoken==0.7.0
tokenizers==0.19.1
torch==2.3.1+cu118
torchaudio==2.3.1+cu118
torchvision==0.18.1+cu118
tqdm==4.66.4
transformers==4.42.3
typing_extensions==4.9.0
urllib3==2.2.2
whisper-timestamped==1.15.4

My python code:

import whisper_timestamped as whisper

audio = whisper.load_audio("test.mp3")
model = whisper.load_model("model", device="cuda")
result = whisper.transcribe(model, audio, language="de")

import json
with open('data.json', 'w', encoding='utf-8') as f:
    json.dump(result, f, ensure_ascii=False, indent=4)

Debug:

(venv) C:\Users\test\Desktop\whisper2>python whisper_test.py
100%|██████████████████████████████████████████████████████████████████████████| 5600/5600 [03:12<00:00, 29.03frames/s]

(venv) C:\Users\test\Desktop\whisper2>python whisper_test.py
 54%|███████████████████████████████████████                                  | 3000/5600 [00:14<00:12, 200.77frames/s]DEBUG:whisper_timestamped:Add segment 1 at step 114:
        <|transcribe|><|0.00|>Also gut, sagte die Denkmaschine und erhob sich. Sagen wir, die Todeszelle im Chisholm-Gefängnis. Die Todeszelle im Chisholm-Gefängnis. Und was werden sie anziehen? So wenig wie möglich, sagte die Denkmaschine. Schuhe, Strümpfe, Hose und ein Hemd. Sie werden sich natürlich durchsuchen lassen. Ich werde exakt so behandelt, wie alle Gefangenen behandelt werden.
DEBUG:whisper_timestamped:WARNING: end timestamp not produced. Adding <|endoftext|>
DEBUG:whisper_timestamped:Missing end token in <|0.00|>Also gut, sagte die Denkmaschine und erhob sich. Sagen wir, die Todeszelle im Chisholm-Gefängnis. Die Todeszelle im Chisholm-Gefängnis. Und was werden sie anziehen? So wenig wie möglich, sagte die Denkmaschine. Schuhe, Strümpfe, Hose und ein Hemd. Sie werden sich natürlich durchsuchen lassen. Ich werde exakt so behandelt, wie alle Gefangenen behandelt werden.<|endoftext|>
100%|█████████████████████████████████████████████████████████████████████████| 5600/5600 [00:44<00:00, 126.38frames/s]
DEBUG:whisper_timestamped:Add segment 2 at step 337:
        <|transcribe|><|0.00|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|>
DEBUG:whisper_timestamped:WARNING: decoding hit the max limit for segment [50360, 50365, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261] (It usually happens when the language model gets stuck)
DEBUG:whisper_timestamped:         Guess last token using probas (assuming greedy decoding): 50261
DEBUG:whisper_timestamped:WARNING: also add last token: <|de|>
DEBUG:whisper_timestamped:Missing end token in <|0.00|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|>
WARNING:whisper_timestamped:Inconsistent number of segments: whisper_segments (1) != timestamped_word_segments (2)
Traceback (most recent call last):
  File "C:\Users\test\Desktop\whisper2\whisper_test.py", line 12, in <module>
    result = whisper.transcribe(model, audio, language="de")
  File "C:\Users\test\Desktop\whisper2\venv\lib\site-packages\whisper_timestamped\transcribe.py", line 296, in transcribe_timestamped
    (transcription, words) = _transcribe_timestamped_efficient(model, audio,
  File "C:\Users\test\Desktop\whisper2\venv\lib\site-packages\whisper_timestamped\transcribe.py", line 920, in _transcribe_timestamped_efficient
    assert l1 == l2 or l1 == 0, f"Inconsistent number of segments: whisper_segments ({l1}) != timestamped_word_segments ({l2})"
AssertionError: Inconsistent number of segments: whisper_segments (1) != timestamped_word_segments (2)

As mentioned in other issues:

naive_approach = True

To make it work at all, but then only a part of the audio is transcribed.

[Jeronymous]

Thanks a lot @dominikmau
Somehow I missed this message during the summer, but what you give is super useful to reproduce and understand/solve the issue (thanks for the professional issue reporting 🙏).

Only problem : the link to the audio is broken. Probably because I come here too late... Can you please resend (if it's a small file, it can be enclosed here, by zipping and drag&dropping the zip file in this discussion)

[yurkoturskiy]

Hi @Jeronymous

I experience the same Inconsistent number of segments issue. I have a lot of assets and happy to share them privately. Can I use your linagora.com email to send the link?

[Jeronymous]

Yes @yurkoturskiy ! Thanks
A few examples to reproduce should be enough.
The shorter the better :)

[yurkoturskiy]

Great! I've made a folder and shared the link over email. Please, let me know if there is anything I can do to help to replicate and fix it.
Thank you!

[ahmedHanzala]

Did we manage to find a fix for this?

[hwscut]

anyone fix this bug? also get same error

[SLbaba]

I am encountering a similar issue when running the batch ['fe_03_10495', 'fe_03_08817', 'fe_03_06312', 'fe_03_01476'](Fisher Eval Datasets). The error message is as follows:

AssertionError: Inconsistent number of segments: whisper_segments (323) != timestamped_word_segments (321)
assert l1 == l2 or l1 == 0, f"Inconsistent number of segments: whisper_segments ({l1}) != timestamped_word_segments ({l2})"

[Jeronymous]

@hwscut @SLbaba what version of openai-whisper do you have ?

pip index versions openai-whisper

I recently saw cases that were failing with openai-whisper==20230314 which were all solved by using the latest version of openai-whisper

pip install openai-whisper==20240930

So I am wondering if upgrading whisper like this would not be the solution to this annoying bug.

(it would not be the case if one of you observe the failure with the latest version 20240930)

[yurkoturskiy]

I'll evaluate the latest whisper, but as you said, it might not be a solution. I'll let you know how it goes.

[SLbaba]

[Jeronymous]

Thank you @SLbaba
can you please give the exact thing to reproduce (all options of command line or python transcribe() function)?

['fe_03_10495', 'fe_03_08817', 'fe_03_06312', 'fe_03_01476'](Fisher Eval Datasets).

Can you also please indicate where to find those audios ?