Endless loop

[doublex]

python 3.10 on GPU fails, but CPU works

Input: input.zip
Testcase:

import whisper
model = whisper.load_model("small")
for filepath in ['input1.wav','input2.wav']:
    whisper.transcribe( model, filepath, language='de', condition_on_previous_text=False, temperature=[0.0], verbose=True )

Output:

[00:00.000 --> 00:07.000]  Und nun ist es Zeit für die wissenschaftliche Gemeinschaft zuzugeben, dass wir bei Covid
[00:07.000 --> 00:14.000]  und ja, den Covid falsch gelegen haben und das ganze Menschenlegen gekostet hat.
[00:14.000 --> 00:21.000]  Wie gesagt, viele, viele, die hier einfach mitgemacht haben, haben wohl nicht entsprechend nachgedacht.
[00:21.000 --> 00:23.000]  das gilt für 90%.
[00:00.000 --> 00:06.440]  der Menschen, die eben hier mitgemacht haben, allerdings sehe ich sehr wohl bei 10 Prozent
[00:06.440 --> 00:12.240]  eine gewisse Boshaftigkeit auch hier und gewisse Pläne hier tatsächlich. Ja, das, was auf den
[00:12.240 --> 00:12.240]  Georgia-Gy
[00:12.240 --> 00:12.240]  Georgia-Gy
[00:12.240 --> 00:12.240]  Georgia-Gy
[00:12.240 --> 00:12.240]  Georgia-Gy
[00:12.240 --> 00:12.240]  Georgia-Gy
[00:12.240 --> 00:12.240]  Georgia-Gy
[....]

[Jeronymous]

I have the same problem, of Whisper getting stucked in an infinite loop, with the latest version "20230124" after this fix

With this waveform.pkl here is the minimal code to reproduce (for me it "fails" both on CPU and GPU)

import pickle
import whisper
waveform = pickle.load(open("waveform.pkl","rb"))
model = whisper.load_model("base")
model.transcribe(waveform, language="fr", task="transcribe", beam_size=5, condition_on_previous_text=False)

The infinite loop happens here in transcribe.py

175:         while seek < num_frames:
...
...
181:             result: DecodingResult = decode_with_fallback(segment)
182:             tokens = torch.tensor(result.tokens)
...
...
195:             timestamp_tokens: torch.Tensor = tokens.ge(tokenizer.timestamp_begin)
196:             consecutive = torch.where(timestamp_tokens[:-1] & timestamp_tokens[1:])[0].add_(1)
197:             if len(consecutive) > 0:  # if the output contains two consecutive timestamp tokens
...
...
214:                 last_timestamp_position = (
215:                     tokens[last_slice - 1].item() - tokenizer.timestamp_begin
216:                 )
217:                 seek += last_timestamp_position * input_stride
218:                 all_tokens.extend(tokens[: last_slice + 1].tolist())

at line 215, last_timestamp_position is 0.
It happens because the list of predicted tokens is <|0.00|> et puis alors qu'est ce qu'il se passe effectivement sur les<|pt|><|translate|><|0.00|><|pt|><|transcribe|><|0.00|><|pt|><|translate|><|0.00|><|1.90|> (then tokens[last_slice - 1] happens to be the last <|0.00|>).
Then it start decoding the exact same thing over and over (because condition_on_previous_text is False).

There should be a safeguard to avoid seek from staying at 0.

Also I wonder why tokens[last_slice - 1] at line 215, and not tokens[last_slice] nor tokens[last_slice - 1: last_slice + 1].max().

[Jeronymous]

Suggestion for a fix in #914 (review)

[Jeronymous]

@jongwook Is somebody working on this issue?
Possibility of an infinite loop is quite a nasty issue.

[FernanOrtega]

I also suffered from this endless loop problem.

[swalker234]

I am experiencing it too but it seems only on large v1 and v2 models

[nfranke]

I've had this happen on medium.en where it was using plenty of CPU and GPU, but never finished transcribing even after 30 minutes for an 11 second file.

[getrasa]

I'm also getting endless loops in the API whenever audio starts with a short segment of music.