CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions

We demonstrate that carefully adjusting the tokenizer of the Whisper speech recognition model significantly improves the precision of word-level timestamps when applying dynamic time warping to the decoder’s cross-attention scores. We fine- tune the model to produce more verbatim speech transcriptions and employ several techniques to increase robustness against multiple speakers and background noise. These adjustments achieve state-of-the-art performance on benchmarks for verba- tim speech transcription, word segmentation, and the timed de- tection of filler events, and can further mitigate transcription hallucinations. The code is available open source.