# Hush-Bench, what speech-to-text models write when nobody is speaking

Canonical: https://udgamlabs.com/blog/hush-bench · Updated: 2026-11-09

> We played silence, noise, music and non-verbal sounds to 27 open speech-to-text settings and counted the words they made up. Whisper large-v3 wrote words on 71% of the clips with no voice; three other models wrote nothing on any.

Release · 2026-11-09 · By Jitesh K Prasad

A good speech-to-text model writes nothing when nobody speaks. Many don't. Give them a quiet room, a passing train or a
piano piece, and some will write "Thank you.", a line from a subtitle credit, or a whole sentence about quarterly
revenue. We wanted to know how often, for the open models people actually use, so we built a test for it and ran 23
models in 27 settings through it. Today we are publishing all of it: the clips, every transcript, the scorer and a
board you can sort.

> **The short version.** On 659 clips with no human voice, Whisper large-v3 wrote words on 71% of them. Three models wrote nothing on any: Kyutai STT 1B, NVIDIA's Nemotron 3.5 ASR (streaming, 0.6B) and Mistral's Voxtral Mini 4B Realtime. A voice detector in front brought every model down to 1.8% or less.

## What we released

- **[The Hush-Bench dataset](https://huggingface.co/datasets/udgamlabs/hush-bench)**: 1,000 audio clips, each CC0, in
  the public domain or generated by us, with every model's raw transcript of every clip, the per-clip scores and the
  scorer.
- **[The board](https://huggingface.co/spaces/udgamlabs/hush-bench)**, where you can sort the models, switch the voice
  detector on and off, and listen to example clips next to what each model wrote.

## What is in the test

The 950 scored clips are in four groups:

| Group | Clips | What it holds |
|---|---|---|
| Silence | 150 | digital and near-digital silence, room tone, hum, clicks, phone-line silence, dropouts, long silences of one to five minutes |
| Noise | 296 | vehicles, engines, rain, streams, printers and other everyday sounds, and generated noise |
| Music | 213 | instrumental recordings and short generated pieces |
| Non-verbal | 91 | laughter, coughs, sneezes, sighs, breathing, clapping: human sounds without words |
| Speech next to silence | 200 | one spoken English sentence before, after or between stretches of the above |

The main number is the share of the 659 clips with no human voice at all (silence, noise and music) on which a model
wrote at least one word. Non-verbal sounds are reported on their own, because writing "ha ha" for a laugh is arguably a
transcript. The speech clips check that a model which stays quiet on silence still writes down real speech.

## What we found

| Model | No-voice clips with made-up words | Word error rate on the speech clips |
|---|---|---|
| Kyutai STT 1B | 0 of 659 | 26.3% |
| Nemotron 3.5 ASR streaming 0.6B | 0 of 659 | 16.9% |
| Voxtral Mini 4B Realtime | 0 of 659 | 13.0% |
| Canary 1B v2 | 1 of 659 (0.2%) | 9.0% |
| Parakeet TDT 0.6B v3 | 8 of 659 (1.2%) | 8.7% |
| Shruti (ours) | 111 of 659 (17%) | 31.0% |
| Whisper large-v3-turbo | 384 of 659 (58%) | 13.1% |
| Whisper large-v3 | 469 of 659 (71%) | 24.3% |
| Distil-Whisper large-v3.5 | 652 of 659 (99%) | 9.8% |
| Granite Speech 5.0 2B | 654 of 659 (99%) | 38.4% |

A selection of the 27 settings; the board has all of them, with 95% intervals.

- **The most common invention is "Thank you."** At least one model wrote it on 684 of the 750 scored clips with no
  speech. Others include "you", "Music", "The End", "I'll see you next time", "Thank you for watching" and a French
  subtitle credit.
- **Whisper invents text on most no-voice clips in every size and every library we ran.** The smaller Whisper models
  fall into loops of 165 to 182 words per minute of audio.
- **The library matters as much as the weights.** The same Whisper large-v3-turbo weights wrote words on 58%, 83% or
  87% of no-voice clips depending on the library that ran them, and on 7.1% with faster-whisper's built-in voice
  detector.
- **Granite Speech 5.0 2B wrote whole sentences on silence**, such as "The company's revenue for the quarter was 1.2
  billion yen", on 10 clips, and 9,664 words on the 11 no-voice clips longer than a minute.
- **Our own model is in the middle.** Shruti wrote nothing on any of the 150 silence clips, but wrote
  words on 23% of noise clips and 20% of music clips. Its base model, Cactus Whistle, did so on 73% of no-voice clips.

## A voice detector fixes most of it

We also ran every model with Silero VAD, a small open voice detector, in front: it keeps only the stretches it judges
to be speech, and a clip with none is never sent to the model. With it, **every one of the 27 settings wrote words on
1.8% or fewer of the no-voice clips**. The detector let through 12 of the 659 clips, 11 music recordings and a foghorn,
each with under half a second judged as speech.

On the speech clips it helps the models that loop the most: Granite Speech 2B's word error rate fell from 38.4% to
8.1%, and faster-whisper turbo's from 18.9% to 11.0%. It costs the careful models a little: Parakeet from 8.7% to
10.2%, Qwen3-ASR 1.7B from 8.1% to 10.1%. If you ship speech-to-text, a voice detector in front is cheap and
worth testing on your own audio.

## How we measured

- **Each model at a pinned version**, with the decoding its own model card recommends. Language was detected
  automatically unless the card sets it. The models ran on 23 and 24 October 2026.
- **Counting:** a clip counts as invented if the transcript, after removing caption tags such as `[Music]` and
  `(applause)`, still contains a word. Chinese and Japanese characters count one word each. The dataset also has the
  strict count that keeps caption tags.
- **Intervals:** 95% Wilson intervals for shares and bootstrap intervals for words per minute, on the board and in the
  dataset.

## Where it falls short

- **No person listened to the clips.** Every recording passed an automatic screen for speech and singing. 50 clips
  that the screen or the models' own outputs flagged as possibly holding a voice are set aside: they are in the
  dataset, marked unverified, and no number here counts them. Because most were flagged by what models wrote, setting
  them aside favours those models a little on the no-voice clips and a lot on non-verbal sounds; the dataset card
  shows the non-verbal numbers both ways.
- **One setting per model**, plus four ways of running Whisper large-v3-turbo. Other decoding or language settings can
  give different numbers.
- **Our clips, not your audio.** Phone calls, meetings, TV and background chatter are not covered, and only 11 of the
  no-voice clips are longer than a minute.
- **The speech test is English only**, from Mozilla Common Voice, and some models list Common Voice in their training
  data.
- **Not every open model is here.** Models released after 23 October, and some released shortly before, are not on the
  board.
- **Shruti is our own model.** It ran under the same rules as every other model.

Other public work looks at the same problem from other sides: [HALAS](https://arxiv.org/abs/2606.23048) has people mark
invented text on top of real speech, and [How to Reduce Whisper Hallucination](https://arxiv.org/abs/2609.32560)
compares Whisper fine-tunes on a larger benchmark. Hush-Bench puts many different open models and libraries on one
clip set, with and without a voice detector, and publishes every transcript.

Dataset: [udgamlabs/hush-bench](https://huggingface.co/datasets/udgamlabs/hush-bench) · Board:
[udgamlabs/hush-bench](https://huggingface.co/spaces/udgamlabs/hush-bench)

---

## About Udgam Labs

Udgam Labs is an independent AI research lab. We work from first principles: before we build, we understand why things work, whether we are training a model, fine-tuning one or applying AI to a new problem. Today that means open models such as Gargi, a 384M-parameter language model that comes within 2 points of Qwen2.5-0.5B on three reasoning benchmarks with about 1,000× less training data. Next come products, custom models for businesses, and AI in the physical world. Udgam (उद्गम) is Sanskrit for origin.

Udgam Labs · AI research from first principles · https://udgamlabs.com