Sumanth (@Sumanth_077) “Accurate STT is not enough for reliable voice agents! Modern speech-to-text is a” — TopicDigg

Accurate STT is not enough for reliable voice agents! Modern speech-to-text is already very good at handling things like traffic, keyboards, air conditioning, and other background noise. Another human voice is a different problem. If someone nearby starts talking while you are speaking, the STT model is no longer trying to separate speech from noise. It is receiving two valid speech signals and has to decide which one actually belongs to the user. For a voice agent, that can become much more than a transcription error. The wrong speaker can change the perceived intent, trigger a turn the user never intended, or influence what the agent does next. I went through Krisp's public Voice Isolation benchmark to see how much difference the audio layer makes before anything reaches the STT model. Across 265 real recordings and 11 STT configurations, pooled word error rate dropped from 23.3% on raw audio to 6.2% after Voice Isolation. The environment breakdown makes the result more useful: - Work: 31.8% → 6.3% WER - Call Center: 23.9% → 6.9% WER - Phone Calls: 3.5% → 3.9% WER That last one matters. When there is very little competing speech, Voice Isolation does not magically improve everything. In some cases it can slightly hurt transcription. The gains show up where the second-speaker problem actually exists. And I think that is the more important point for voice agents. There are really two questions: Did we transcribe the words correctly? Did we transcribe the right person? For a voice agent that can call tools, update memory, retrieve information, or take actions, the second one can be just as important as the first. The pipeline really starts before STT: Microphone → Voice Isolation → Speech-to-Text → Turn Detection → Agent → Tools → Response A stronger LLM later in the stack cannot recover the user's original intent if the wrong speaker already made it into the transcript. I’ve shared the benchmarks below