Sauti · Methodology
How we score, and why you can trust it
A leaderboard is only worth anything if it's fair and reproducible. Here is exactly how results are produced — and the choices that stop it from being gamed or gamed-into.
One fixed, versioned text normalizer
Text normalization is the single biggest lever on word error rate — raw vs fully-normalized can swing ~10 absolute points, larger than the gap between competing models. Every model's output and every reference pass through the same frozen normalizer (currently norm-v1.0). Changing it ships a new version and triggers a full re-run; we never silently mutate a version.
Human-verified references
Ground-truth transcripts are human-corrected, not machine-generated. A wrong reference yields a wrong WER while the engine works perfectly — so reference quality is treated as the foundation, with inter-annotator checks on a subset.
A private held-out set
A portion of every dataset is kept private and never published. Models are evaluated on it on a schedule; only aggregates are released. If a model's public score far exceeds its held-out score, that gap is the tell for overfitting or contamination. We publish both numbers side by side on the leaderboard so that check is one you can run rather than one you have to take our word for. The held-out clips themselves stay unpublished, which is also why we do not fold them into the headline to get tighter intervals: that would buy precision once and end the check for good.
Immutable, versioned datasets
Each dataset version is frozen — audio and references never change after release. Results always carry the dataset version, normalizer version, and run date, so any number can be reproduced exactly.
Confidence intervals, not false precision
Aggregate WER is computed corpus-style (total edits ÷ total reference words), and every score ships a bootstrap 95% confidence interval (seeded, so it's reproducible). When two models' intervals overlap we mark them a tie (≈) — we don't claim a rank gap the data can't support.
Closed APIs are pinned and snapshotted
Commercial APIs change silently. We pin the exact model version, snapshot every raw response, and treat open (downloadable weights) and closed (API) models as distinct classes. Speed (real-time factor) is only compared on identical, controlled hardware — never API-vs-local.
Per-language and per-dialect breakdowns
Aggregate WER hides disparities. Results are broken out by language, dialect, accent, and domain — which is how the African track exposes gaps a global-English number would bury.
Open methodology
The normalizer and scoring engine are open-source, and model transcripts are released for audit. Transparency invites scrutiny; opaque benchmarks get gamed.