OpenAI Whisper glitches - repeated lines and timing off

[Windrider6]

I'm using OpenAI Whisper in Subtitle Edit, and it is glitching. Lines will be repeated several times, sometimes a few lines repeated. This will throw the timing out, and lines of dialog will be missed. This is also happening with Faster Whisper.

Anyone else see this? Is there a solution to the problem?

[niksedk]

Hopefully this will improve in a future version of Whisper.

You can try to experiment with some of the parameters, like "--beam_size" (OpenAI) or "--max-context" (CPP/Const-me).

A parameter (the Advanced button) can be added when using latest SE beta: https://github.com/SubtitleEdit/subtitleedit/releases/download/3.6.13/SubtitleEditBeta.zip

[Windrider6]

OK. Questions:

1. What is the default beam size value and the range of values?
2. What would you recommend the beam size be for a Windows 10 computer, 3.7 GHz processor and 32 GB of RAM?

[niksedk]

I'm not sure what the range is, probably around -1 to 15... try with something like 1-3.
--beam_size BEAM_SIZE number of beams in beam search, only applicable when temperature is zero (default: 5)

Here is another parameter you can try:
--best_of BEST_OF number of candidates when sampling with non-zero temperature (default: 5)

[Windrider6]

Thank you. I read "Introduction to Beam Search Algorithm", but cannot understand it.
What does changing the beam size actually do in practical matters like processing power, RAM, and processing time, when the number is lower or higher than the default?
Ug. What the heck is "temperature" supposed to be? It's above freezing in here ...

[Windrider6]

I've tried several options now, and all of them have produced the repeating lines glitch:
CTranslate2 (Faster Whisper): medium (1.5 GB)
--beam_size 1
CTranslate2 (Faster Whisper): large (2.9 GB):
--beam_size 1
--beam_size 15
--temperature 0 --beam_size 10 --patience 2
--temperature 0.2 --best_of 10 --patience 2

I clipped out a 1 minute section of the TV show where the last test had the glitch, then ran the process with the same parameters, and did not get the glitch.

I don't know what to do now.

[Purfview]

Is it stuck in repeating lines or just repeats some lines and then continues normally?
Try Standalone Faster Whisper.

Would be helpful if you could share an audio sample to reproduce the issue.

[Purfview]

Is it possible to use Standalone Faster Whisper to produce subtitles that are timed properly?

Yes. Check if you get this glitch with it.

[Windrider6]

I give up. I don't understand how to use Python. I tried following the instructions at https://github.com/guillaumekln/faster-whisper, but it will not work in python console or as a py file.

I really want to create subtitles for some TV shows that do not have any available, but the number of problems I am encountering are just too much for me.

[Purfview]

Why you are going to another repo if I posted you this link -> https://github.com/Purfview/whisper-standalone-win
In releases you will find ready to run exe. you don't need Python at all.

PS:
You can run it in console or in Subtitle Edit, Btw, latest Subtitle Edit beta has option to download it automatically.

[Windrider6]

I misunderstood.

Used Standalone Faster Whisper and did not get those repeated lines glitch. Great! Why is that? How is it different?

Fixing the word errors and timing still takes me a long time.

[Purfview]

Because OpenAI's pseudo VAD is not good.
Of course you need to spend some time in fixing subtitles, model is not 100% accurate.