A dental scheduling app added a voice assistant last spring, and the demo went perfectly. Then a patient called from her kitchen with the dishwasher running, said “Thursday the fourth,” and got booked for the fourteenth. She showed up ten days late to find her slot long gone.
The model wasn’t broken. Nobody had ever tested it against a kitchen.
Pick the Architecture Before You Pick the Voice
Most AI voice agents run as a chain. Speech recognition turns audio into text, a language model decides what to say, and a synthesis engine reads the reply back. Some newer systems collapse those steps into a single speech-to-speech model, which sounds more natural and responds faster. It’s also harder to debug. When a booking goes wrong, you can’t open a transcript and see whether the error came from hearing or reasoning.
For most production apps, the chain is the better starting point. You can swap one component without touching the others, log every stage, and write business rules against text you can actually read. Teams that start with the all-in-one model often rebuild the chain anyway once their compliance team asks for call records.
Latency Shows Up Before Accuracy Does
Callers forgive a wrong answer faster than a long pause. After roughly 800 milliseconds of silence, people start repeating themselves, and now the agent is processing two overlapping requests.
Budget the delay by stage. Recognition might take 200 milliseconds, the model 400, synthesis another 150 before the first syllable plays. Streaming helps at every step, so start speaking the first sentence while the model is still writing the second.
Endpointing deserves its own tuning. This is the logic that decides a person has finished talking. Set it too aggressive and the agent cuts off anyone who pauses to check their calendar. Too cautious, and every exchange drags.
Benchmark With Your Own Audio
A public speech-to-text benchmark tells you how a model handles clean read speech, audiobooks, or broadcast news. Your users are calling from cars.
Build a test set from real recordings, collected with consent, that covers the conditions you actually see: speakerphone, accents common in your market, barking dogs, people spelling their last names letter by letter. Two hundred labeled clips is enough to start.
Then measure more than word error rate. A model that gets 96 percent of words right can still miss the 4 percent that carry meaning, like dates, dollar amounts, and medication names. Tag those entities and score them separately. The dental app’s overall error rate looked excellent. Its accuracy on spoken dates did not.
Rerun your own speech-to-text benchmark every time a vendor updates a model. Gains on general conversation sometimes arrive with regressions on short numeric phrases, and the release notes won’t mention it.
Test the Conversation, Too
Transcription is one layer. The full agent needs scripted call scenarios: a caller who changes their mind halfway through, one who interrupts the greeting, one who goes silent for six seconds. Simulated callers can run hundreds of these overnight.
Still, listen to a sample by hand each week. Automated scores miss the moment an agent sounds curt or confused.
Track task completion as well. Did the appointment land in the calendar with the right date and the right patient? That single number says more about the product than any transcript metric.
A Final Thought
The dental app’s fix took one afternoon: a confirmation step that read the date back and waited for a yes. The hard part was realizing they needed it, and that only came from listening to calls their tests had never imagined.
