Why is GGML so much faster than PyTorch?

[wizardforcel]

Test data on my computer, with TR-3970x and RTX3080Ti

Whisper Medium+PyTorch CPU:

Takes 3 hours for audio per hour

Whisper Medium+PyTorch GPU:

10 min per hour

Whisper Large V2+GGML CPU:

30 min per hour

[gordicaleksa]

1. Are you sure you're running the models with the same hyperparams? (beam size, etc.)
2. GGML is bare bone, "close to metal" (written in C), hence more optimal but less developer friendly which is a necessary tradeoff Georgi had to make

[ggerganov]

Yes, it's most likely due to different beam size, greedy vs beam-search decoder, etc.
If you match the parameters 1-to-1 then it is not much faster. Should be comparable