A good speech-to-text model writes nothing when nobody speaks. Many don't. Give them a quiet room, a passing train or a piano piece, and some will write "Thank you.", a line from a subtitle credit, or a whole sentence about quarterly revenue. We wanted to know how often, for the open models people actually use, so we built a test for it and ran 23 models in 27 settings through it. Today we are publishing all of it: the clips, every transcript, the scorer and a board you can sort.
What we released
- The Hush-Bench dataset: 1,000 audio clips, each CC0, in the public domain or generated by us, with every model's raw transcript of every clip, the per-clip scores and the scorer.
- The board, where you can sort the models, switch the voice detector on and off, and listen to example clips next to what each model wrote.
What is in the test
The 950 scored clips are in four groups:
| Group | Clips | What it holds |
|---|---|---|
| Silence | 150 | digital and near-digital silence, room tone, hum, clicks, phone-line silence, dropouts, long silences of one to five minutes |
| Noise | 296 | vehicles, engines, rain, streams, printers and other everyday sounds, and generated noise |
| Music | 213 | instrumental recordings and short generated pieces |
| Non-verbal | 91 | laughter, coughs, sneezes, sighs, breathing, clapping: human sounds without words |
| Speech next to silence | 200 | one spoken English sentence before, after or between stretches of the above |
The main number is the share of the 659 clips with no human voice at all (silence, noise and music) on which a model wrote at least one word. Non-verbal sounds are reported on their own, because writing "ha ha" for a laugh is arguably a transcript. The speech clips check that a model which stays quiet on silence still writes down real speech.
What we found
| Model | No-voice clips with made-up words | Word error rate on the speech clips |
|---|---|---|
| Kyutai STT 1B | 0 of 659 | 26.3% |
| Nemotron 3.5 ASR streaming 0.6B | 0 of 659 | 16.9% |
| Voxtral Mini 4B Realtime | 0 of 659 | 13.0% |
| Canary 1B v2 | 1 of 659 (0.2%) | 9.0% |
| Parakeet TDT 0.6B v3 | 8 of 659 (1.2%) | 8.7% |
| Shruti (ours) | 111 of 659 (17%) | 31.0% |
| Whisper large-v3-turbo | 384 of 659 (58%) | 13.1% |
| Whisper large-v3 | 469 of 659 (71%) | 24.3% |
| Distil-Whisper large-v3.5 | 652 of 659 (99%) | 9.8% |
| Granite Speech 5.0 2B | 654 of 659 (99%) | 38.4% |
A selection of the 27 settings; the board has all of them, with 95% intervals.
- The most common invention is "Thank you." At least one model wrote it on 684 of the 750 scored clips with no speech. Others include "you", "Music", "The End", "I'll see you next time", "Thank you for watching" and a French subtitle credit.
- Whisper invents text on most no-voice clips in every size and every library we ran. The smaller Whisper models fall into loops of 165 to 182 words per minute of audio.
- The library matters as much as the weights. The same Whisper large-v3-turbo weights wrote words on 58%, 83% or 87% of no-voice clips depending on the library that ran them, and on 7.1% with faster-whisper's built-in voice detector.
- Granite Speech 5.0 2B wrote whole sentences on silence, such as "The company's revenue for the quarter was 1.2 billion yen", on 10 clips, and 9,664 words on the 11 no-voice clips longer than a minute.
- Our own model is in the middle. Shruti wrote nothing on any of the 150 silence clips, but wrote words on 23% of noise clips and 20% of music clips. Its base model, Cactus Whistle, did so on 73% of no-voice clips.
A voice detector fixes most of it
We also ran every model with Silero VAD, a small open voice detector, in front: it keeps only the stretches it judges to be speech, and a clip with none is never sent to the model. With it, every one of the 27 settings wrote words on 1.8% or fewer of the no-voice clips. The detector let through 12 of the 659 clips, 11 music recordings and a foghorn, each with under half a second judged as speech.
On the speech clips it helps the models that loop the most: Granite Speech 2B's word error rate fell from 38.4% to 8.1%, and faster-whisper turbo's from 18.9% to 11.0%. It costs the careful models a little: Parakeet from 8.7% to 10.2%, Qwen3-ASR 1.7B from 8.1% to 10.1%. If you ship speech-to-text, a voice detector in front is cheap and worth testing on your own audio.
How we measured
- Each model at a pinned version, with the decoding its own model card recommends. Language was detected automatically unless the card sets it. The models ran on 23 and 24 October 2026.
- Counting: a clip counts as invented if the transcript, after removing caption tags such as
[Music]and(applause), still contains a word. Chinese and Japanese characters count one word each. The dataset also has the strict count that keeps caption tags. - Intervals: 95% Wilson intervals for shares and bootstrap intervals for words per minute, on the board and in the dataset.
Where it falls short
- No person listened to the clips. Every recording passed an automatic screen for speech and singing. 50 clips that the screen or the models' own outputs flagged as possibly holding a voice are set aside: they are in the dataset, marked unverified, and no number here counts them. Because most were flagged by what models wrote, setting them aside favours those models a little on the no-voice clips and a lot on non-verbal sounds; the dataset card shows the non-verbal numbers both ways.
- One setting per model, plus four ways of running Whisper large-v3-turbo. Other decoding or language settings can give different numbers.
- Our clips, not your audio. Phone calls, meetings, TV and background chatter are not covered, and only 11 of the no-voice clips are longer than a minute.
- The speech test is English only, from Mozilla Common Voice, and some models list Common Voice in their training data.
- Not every open model is here. Models released after 23 October, and some released shortly before, are not on the board.
- Shruti is our own model. It ran under the same rules as every other model.
Other public work looks at the same problem from other sides: HALAS has people mark invented text on top of real speech, and How to Reduce Whisper Hallucination compares Whisper fine-tunes on a larger benchmark. Hush-Bench puts many different open models and libraries on one clip set, with and without a voice detector, and publishes every transcript.
Dataset: udgamlabs/hush-bench · Board: udgamlabs/hush-bench