How good is Thai speech-to-text, really?
Nobody had measured it across the tools you’d actually reach for — commercial APIs and open models, on the same Thai audio, scored the same way. So we did. This is the first board.
The models best at Thai drop every English word
Every model that nails Thai characters throws away the English words mixed into the speech. Only plain Whisper keeps them, and it pays with slightly worse Thai. The chart plots those two against each other — the best corner, top-right, sits empty.
Five models, same clips, same scorer
CER is lower-is-better; English retention is higher-is-better; speed is the realtime factor on an L4 GPU (commercial APIs over the network). Every number reruns from the repo.
| Model | Type | CER ↓ | Keeps English ↑ | Speed |
|---|---|---|---|---|
| Thonburian large-v3 | open | 6.9% | 0% | 6.0× |
| Whisper large-v3 | open | 9.0% | 50% | 6.0× |
| Typhoon ASR | open | 9.3% | 0% | 1.2× |
| Deepgram Nova-3 (th) | commercial | 9.7% | 0% | 3.8× |
| Deepgram Nova-3 (multi) | commercial | 99.3% | 25% | 14.4× |
CER, because Thai has no spaces
Why not word error rate
Thai writes without spaces between words, so word error rate needs a tokenizer just to decide where one word ends — and every paper picks a different one. Character error rate needs no tokenizer, so a model can’t lower its score by choosing a lenient one.
The English-retention number
Thai speakers drop English words mid-sentence. A single CER hides whether the model caught “meeting” or replaced it with something Thai-shaped, so we also report how many of the reference’s English tokens survived.
One normalization pass sits between every model and its grade — NFC, lowercase, Thai digits to Arabic, spaces and punctuation dropped — the same pass for all of them. A real clip a model returned:
แจเนี่ยท่านผู้เชี่ยวชาญด้านโรคภัยประจําสหประชาชาติแถลงว่า เป็นไปได้ที่การระบาดของโรคในครั้งนี้เกิดจากเจ้าหน้าที่…
What this board is not — yet
- Six clips of clean read speech. This is FLEURS Thai, not real conversation, so the exact numbers will shift as we add data. The overall pattern is what holds.
- The hard case isn’t here. Real conversational Thai with English mixed in needs a licensed code-switch set that doesn’t exist publicly yet. Building it is the main open task.
- Every number reruns from the repo — no hand-edited numbers, no rows we can’t repeat.
# reproduce the whole board bun run fetch # public Thai eval set (FLEURS, CC-BY) modal run scripts/modal_asr.py # open models on a GPU bun run ingest # score -> the board