Sauti · by Wisprs
The honest speech-to-text leaderboard
Honest, reproducible speech-to-text benchmarks — global and African. Every model — open-source and commercial API — is scored against the same human-verified references through one fixed text normalizer, with 95% confidence intervals so a rank gap means something.
Global leaderboard
normalizer norm-v1.0 · 2026-08-08| # | Model | Type | WER (95% CI) | Claimed | CER | Latency | $/hr |
|---|---|---|---|---|---|---|---|
| 1 | grok-stt xai · grok-stt | closed | 9.8%[8.9%–10.9%] | none published | 6.0% | 1465ms | $0.10 |
| 2 | nova-3 deepgram · nova-3 | closed | 11.5%[10.0%–13.1%]≈ | none published | 8.1% | 407ms | $0.46 |
| 3 | gpt-transcribe openai · gpt-transcribe | closed | 11.8%[10.3%–13.6%]≈ | none published | 8.9% | 2039ms | $0.27 |
| 4 | Whisper large-v3 (self-hosted) self-hosted · large-v3 | open | 11.9%[10.4%–13.6%]≈ | none published | 8.9% | 209504ms | $0.00 |
| 5 | universal-2 assemblyai · universal-2 | closed | 12.1%[10.7%–13.9%]≈ | none published | 9.2% | 5811ms | $0.15 |
| 6 | universal-3-5-pro assemblyai · universal-3-5-pro | closed | 12.3%[10.8%–14.0%]≈ | 5.9%vendor-internal | 9.3% | 6365ms | $0.21 |
| 7 | Whisper small (self-hosted) self-hosted · small | open | 12.3%[10.7%–14.0%]≈ | 3.4%LibriSpeech | 9.2% | 44882ms | $0.00 |
| 8 | gpt-4o-mini-transcribe openai · gpt-4o-mini-transcribe | closed | 12.8%[11.4%–14.3%]≈ | none published | 9.7% | 1561ms | $0.18 |
| 9 | whisper-1 openai · whisper-1 | closed | 13.5%[11.6%–15.8%]≈ | none published | 10.1% | 2862ms | $0.36 |
| 10 | scribe_v2 elevenlabs · scribe_v2 | closed | 13.6%[12.3%–15.2%]≈ | none published | 9.0% | 3231ms | $0.22 |
| 11 | scribe_v1 elevenlabs · scribe_v1 | closed | 13.7%[12.4%–15.2%]≈ | none published | 9.0% | 3231ms | — |
| 12 | gpt-4o-transcribe openai · gpt-4o-transcribe | closed | 14.7%[11.9%–18.3%]≈ | none published | 11.7% | 2165ms | $0.36 |
Claimed is the vendor’s own published WER for this language, on the benchmark named next to it (click it for the source). It is comparable to our Measured WER only when that benchmark matches ours — a lower claimed figure on a different or easier benchmark is not a like-for-like win. “none published” means the vendor markets the language but publishes no WER for it.
Cost is each vendor’s list price for pre-recorded pay-as-you-go transcription, read from their own pricing page on 2026-08-08. Streaming rates and committed-volume discounts differ and are not shown, so this column compares list prices, not what you would pay at scale. Self-hosted is marginal cost on hardware we already run, so it excludes capex and power. “—” means the vendor publishes no separate rate for that model, usually because it has been superseded; we leave it blank rather than carry a newer model’s price across. Sources: developers.openai.com, developers.openai.com, deepgram.com, assemblyai.com, elevenlabs.io, elevenlabs.io, docs.x.ai.
WER = word error rate (lower is better). ≈ marks a statistical tie (overlapping confidence intervals). Open = downloadable weights; closed = commercial API (version-pinned).
Dataset: 250 clips (2.9 hours) of real-world, unscripted conversational English speech from People’s Speech (MLCommons, CC-BY), human-transcribed. Scored on a held-out split never used to tune anything. See our State of Podcast Transcription 2026 report for context.
The contamination check
The board above is scored only on the held-out half of the corpus, which we do not publish. This table adds each model’s score on the public half. A model trained on data that includes our public clips would do noticeably better there than on audio it has never seen, so the gap between the two columns is the check. The largest gap on this board is 1.8 points, on whisper-1, which is inside the noise at this sample size. Nothing here looks contaminated.
| Model | Held-out WER (N=77) | Public WER (N=173) | Gap |
|---|---|---|---|
| whisper-1 | 13.5% | 11.6% | +1.8 |
| Whisper small (self-hosted) | 12.3% | 11.7% | +0.6 |
| universal-2 | 12.1% | 11.5% | +0.6 |
| Whisper large-v3 (self-hosted) | 11.9% | 11.3% | +0.6 |
| gpt-transcribe | 11.8% | 11.3% | +0.5 |
| gpt-4o-transcribe | 14.7% | 14.2% | +0.5 |
| gpt-4o-mini-transcribe | 12.8% | 12.3% | +0.5 |
| nova-3 | 11.5% | 11.2% | +0.3 |
| universal-3-5-pro | 12.3% | 12.2% | +0.1 |
| grok-stt | 9.8% | 9.9% | -0.1 |
| scribe_v2 | 13.6% | 14.0% | -0.3 |
| scribe_v1 | 13.7% | 14.0% | -0.3 |
The held-out clips stay unpublished, so this check keeps working as the benchmark gets cited. Held-out N is small and most models on the main board are statistical ties because of it; the fix for that is more audio, not scoring the same audio twice.
Per-language
State of Voice
A quarterly read on where speech AI actually stands — what moved, what regressed, and an honest take on the field, with a focus on multilingual and African real-world audio.
See the African track →