An automated switchboard answered. "Hello, thank you for calling. Press one for billing, two for..." Our agent heard the word "Hello," decided a human had answered, and did what it was built to do: it greeted back. That greeting landed on top of the menu. The IVR, waiting for a keypress, heard a voice instead of a tone and repeated itself. Our agent, still mid-sentence, kept talking. Two automated systems talked over each other for a few seconds, neither one yielding, until the IVR gave up and dropped the line.
That single collision, replayed a few hundred times a day at first, is the difference between a voice agent that works on a phone line and one that quietly burns a client's calling budget for nothing. This piece is about the distance between those two systems: what it took to move ours from "mostly works in testing" to dialing, navigating, and logging real B2B phone calls toward a target of 200,000 a month, and what that actually costs once you strip out the number vendors put on their homepage.
The job and the starting point
The client is a US-based B2B company. The system has to dial real numbers, get past whatever IVR tree answers, tell a human from an answering machine or a hold-music loop, hold a short conversation with whoever picks up, and write structured, accurate results back into the client's CRM. A previous team spent months on it and never reached production. We picked it up in October 2025, four engineers and one person on infrastructure, and reached production in mid-March 2026. Five months. The client's target is 200,000 calls a month; we are currently running several hundred a day and scaling toward it.
On paper this looks like plumbing. Speech to text, a language model, text to speech, wire the three together, point the result at a SIP trunk. That is roughly where we started, and it is roughly where the previous team got stuck. The first two months went into a proof of concept that could dial, listen, and respond. The other three went into the unglamorous part: getting that result to survive an actual phone call, at the accuracy and speed a paying client would sign off on.
Getting the ears right
We went through four speech-to-text engines before we stopped moving: Google STT, then AssemblyAI, then Whisper, and finally Deepgram Nova-3. Of every swap in the stack, this one had the single biggest effect on the system's accuracy.
| Model | WER | CER | Avg latency | Best for |
|---|---|---|---|---|
| Deepgram Nova-3 | 1.53% | 0.57% | ~300ms (streaming) | Production voice agents |
| Groq Whisper Large v3 | 6.82% | 2.23% | ~650ms (batch) | Offline transcription |
Those numbers came from our own benchmark set, and we don't pretend they generalize past it. They're clean test recordings, not the noisy, 8kHz, hold-music-drenched audio our system actually hears on a live call. Deepgram's own published median on diverse enterprise audio is more modest, around 5.26%, and once you push any of these engines into genuinely hostile conditions the picture gets worse and far less stable. One independent 2026 benchmark that scored streaming engines under voice-agent conditions put Deepgram Nova-3's word error rate in the 25 to 30 percent range, with wide run-to-run variance, and a separate comparison found no single engine dominant on both accuracy and latency [2][3]. That spread is the real lesson: published numbers told us almost nothing about our own calls, so we stopped trusting them and benchmarked on our own audio. What the clean-set test did settle, and what held once real hold music and cross-talk arrived, was the choice actually in front of us. On the calls we care about, Deepgram was several times more accurate than Whisper, and the margin widened rather than closed as conditions got worse.
Whisper's specific failures were worse than a bad WER number suggests. On hold music or dead silence it would sometimes hallucinate a plausible-sounding company name that was never said. On voicemail it would occasionally return an empty transcript instead of failing loudly. And Whisper is fundamentally a batch model; we spent close to a month trying to force it into a real-time streaming pipeline and kept hitting chunk-boundary errors that never fully went away. Deepgram is streaming-first: final transcripts land about 300 milliseconds after someone stops talking, with partial transcripts arriving continuously before that. It also accepts a keyword array with probability weighting, so we inject the contact's name and company before every call, which Whisper's prompting can't reliably do since Whisper prompting steers style, not keyword recall.
Cutting ten seconds down to 1.5
There's a known number in conversation-analysis research for how long a natural gap between speakers runs: about 200 milliseconds [1]. Past 800 milliseconds a pause starts to feel wrong. Past a second and a half, most people assume they're talking to a machine, which is exactly the opposite of what a validation call is trying to accomplish. Our first working version of the pipeline had a 5 to 10 second turnaround from "caller stops talking" to "agent starts responding." We got it down to 1.5 to 2.5 seconds in four separate rounds of surgery, and the first bottleneck we found wasn't the one we expected.
Stage one: the retrieval step was the real bottleneck. Our embedding-based lookup, pulling context into the prompt via RAG, was taking one to two seconds per turn and occasionally timing out at three. This is not a quirk unique to us; it's recognized as its own bottleneck class in the literature on voice-agent RAG systems, where the vector round-trip alone can eat most of a latency budget built for a sub-300-millisecond turn [4][5]. We added dedicated embedding pods on more capable nodes and cut that step to roughly 250 milliseconds, which alone took more than a second off the total.
Stage two: swapping the model. We moved from GPT-OSS 120B to Llama 70B, both served on Groq. Llama 70B is genuinely harder to keep on-script; getting it to hold strict phase-based dialogue took more prompt engineering than the bigger model needed. In exchange it runs two to three times faster per token, which roughly halved our LLM latency.
Stage three: replacing a model call with a model, on the pod. Instead of asking the LLM to classify whether a call had hit an IVR, a voicemail, or a live operator, we trained a small BERT classifier and ran it directly on the call-worker pods. No external API round-trip, no waiting on a rate limit. This is also where the fix for our opening scene lives: rather than trying to detect an IVR acoustically, in real time, from tone of voice, the classifier looks at the structure of the transcript itself, and it catches an IVR menu reliably even when the greeting sounds perfectly human.
Voice activity detection turned into its own small saga. FunASR-VAD, our first choice, kept flagging hold music as speech. We swapped to TEN-VAD, which had the identical problem. What actually worked was pairing Silero VAD with Deepgram's own built-in VAD and requiring both to agree that silence had occurred before the agent would respond. Neither one alone was reliable enough; roughly a third of our calls hit hold music or an IVR at some point, so this wasn't an edge case, it was a third of the traffic.
That same noise sensitivity produced a second cascade, quieter than the IVR collision but just as damaging. A bad phone line garbled the greeting on the way in. Our agent, hearing static instead of words, asked the operator to repeat themselves. Generating that request took a beat too long. The operator, hearing silence, repeated the original line on their own, unprompted. Now both sides were talking, out of sync, and by the time the agent caught up the operator had already hung up in frustration. The CRM logged a wrong company name and a wrong call status, and nobody upstream would know it was wrong until someone checked the recording.
Stage four: pre-scripting the obvious. For a small set of common responses, greetings, confirmations, simple acknowledgments, we pre-generate ElevenLabs audio and fire it instantly while the LLM keeps working in the background on whatever comes next. Pre-scripted phrases land in under 500 milliseconds; anything the model has to generate fresh still takes 1.5 to 2.5 seconds. The client's own metric for whether this worked wasn't a latency chart, it was that prospects stopped hanging up in the first three seconds of the call.
| Component | Current | At start |
|---|---|---|
| Deepgram STT | 300ms avg, up to 1s on noisy lines | 500-700ms (Whisper) |
| LLM (Groq) | 500ms median, 700ms mean, 1.2s P90 | 2-3s (GPT-120B) |
| ElevenLabs TTS | ~300ms time to first byte | same |
| RAG embeddings | ~250ms | 1-2s, timing out at 3s |
| Total | 1.5-2.5s | 5-10s |
The bottleneck today is the LLM call, which still accounts for roughly 40 percent of end-to-end latency. We know exactly where the next round of surgery has to happen; we just haven't needed it yet.
We also failed the timeout problem in a way worth admitting. Our first instinct on a slow API response was to play a filler phrase, something like "give me a moment," while we waited. It made things worse. A canned filler line broke the illusion of a live conversation harder than a brief, honest silence did. We pulled it and replaced it with hard timeouts on Groq, Deepgram, and ElevenLabs, a retry on failure, and a hard stop and re-dial past a retry limit.
Why we stayed cascaded
A GPT-4o-Realtime-style speech-to-speech model advertises latency in the 300 to 500 millisecond range under clean, ideal-condition testing. Independent measurements under real network conditions tell a less flattering story: median turn latency stretching well past a second, and into several seconds on longer sessions, numbers that land closer to our own cascaded pipeline than the vendor benchmark suggests [6]. We looked at both the advertised gap and the measured one and stayed cascaded anyway, and the reason isn't nostalgia for the old architecture, it's what the job actually requires.
This system has to navigate multi-level IVR menus with DTMF tones and spelled-out names, run a purpose-built classifier trained on our own call data to catch voicemail versus IVR versus a live human, hold strict phase-based control over where in the conversation the call currently sits rather than leaning on a single system prompt, and write structured fields into a client CRM after every call. A single end-to-end audio model is close to a black box between "audio in" and "audio out." There's nowhere to insert a fine-tuned classifier mid-turn, nowhere to enforce that phase three of the script can't be skipped to phase five, and no clean seam for pulling structured data out afterward. Cascaded architecture is slower, but every stage boundary is a place we can plug in control. We traded raw speed for the ability to guarantee what happens at each step, which mattered more on this job than shaving another few hundred milliseconds off an already-tight budget.
The turn-detection shortcut we didn't take
The more sophisticated approach to knowing when someone has finished speaking uses a language model to judge semantic completeness, not just silence. It's a real improvement in some deployments, and we chose not to build it. Our system already runs a 1.5 to 2.5 second turn budget with every millisecond accounted for, and adding a model call just to decide whether to respond would compete for the same budget we spent four stages clawing back. Our acoustic approach, once we forced Silero and Deepgram's VAD to agree before triggering a response, turned out to be reliable enough for the calls we actually handle. It's a deliberate trade: some conversational nuance in exchange for a latency number we can predict and defend. The semantic option is real, and we may build it once it can earn its place in a budget we fought this hard for.
The bill, line by line
The infrastructure moved from a single AWS instance to Kubernetes, Terraform, and Karpenter autoscaling. We rejected AWS Lambda outright: calls routinely run 10 to 20 minutes and can hit a hard 30-minute ceiling, well past Lambda's 15-minute execution limit, and each call worker needs roughly 2GB of RAM for models, buffers, and conversation state, which Lambda's per-millisecond billing prices worse than a dedicated pod.
The cluster runs on c7i.xlarge instances at $0.1785 an hour. Call-worker nodes scale up to 18, each running four worker pods at 2GB apiece, with Karpenter scaling from zero to fifty active workers based on live call load. Five dedicated embedding nodes avoid memory contention with the call workers. A scheduler node handles the call queue, assignment, and retries; a separate backend node runs the API layer, CRM integration, and post-call processing. At full parallel load the cluster handles roughly 200 calls in a 20-minute window.
Infrastructure cost scales with how hard we're running it: about $1,200 a month sitting idle, $1,600 to $1,800 a month at our current production load of an hour or two of active calling a day, and roughly $7,835 a month if the cluster ran flat out, 24 hours a day. That top figure breaks down into an EKS control plane at $73, admin nodes at $130 each, a 500GB PostgreSQL RDS instance at $261, VPC NAT at $66, a load balancer at $22, and container registry at $20, before the compute itself.
Per minute of actual call time, the real bill looks like this:
| Component | $/min | Notes |
|---|---|---|
| Deepgram STT | $0.0092 | streaming |
| LLM (Groq + OpenAI GPT) | ~$0.31 | roughly 20 calls/min across live conversation and post-call processing |
| AssemblyAI | $0.0035 | post-call QA transcription |
| ElevenLabs TTS | ~$0.10 | generated speech |
| Infra + telephony | ~$0.20 | AWS, SIP, subscriptions |
| Total | ~$0.62 | varies with call complexity |
During a live call we're making roughly ten Llama-70B calls a minute for generation, intent, and retrieval, plus two calls a minute to a larger supervisor model that watches for the agent getting stuck. After the call ends, three GPT calls extract status and write a summary, and another ten Llama calls populate structured fields. Groq prices Llama 70B at $0.59 per million input tokens and $0.79 per million output; each individual call costs close to a cent, but multiply that by twenty-plus calls a minute across an average two-minute conversation and it adds up to about $0.31 a minute on inference alone, the single largest line on the bill.
Set that against what orchestration platforms advertise, typically five to ten cents a minute, and the two numbers look like they're describing different products. They are. The advertised figure covers orchestration only; the underlying model and speech costs still get passed through to you, and none of it includes the infrastructure to run at scale.
Build it or rent it
There's a real set of hosted platforms that solve a version of this problem out of the box: Vapi, Retell, Bland, and Synthflow on the fully managed end, LiveKit and Pipecat if you want an open-source orchestration layer you self-host. For most voice-agent use cases, one of those is the right answer and building your own is a waste of five months.
Ours wasn't most use cases. Custom infrastructure earns its cost when a project genuinely needs all five of: complex IVR and directory navigation with DTMF and name-spelling logic, classification models trained on your own call data rather than a generic one, strict phase-based control over the conversation instead of a single prompt, call volume that clears the point where per-minute platform fees start compounding, and deep, structured integration into a proprietary CRM. Miss two or three of those and a platform will get you there faster and cheaper.
The volume question is worth doing the arithmetic on rather than asserting. Retell publishes roughly $0.07 a minute for orchestration on top of pass-through provider costs; add our own STT, LLM, and TTS figures as a stand-in for those pass-through costs and a Retell-based version of this system lands close to $0.49 a minute, a number that barely moves whether you're running a thousand minutes a month or a million, because the fee is purely per-minute. Our custom stack's $0.62 a minute includes roughly $0.20 of infrastructure that behaves nothing like a per-minute fee: it's closer to fixed, sized for a cluster that can serve up to fifty parallel calls whether five hundred or fifty thousand of those minutes get used this month. At a few hundred calls a day, that infrastructure cost is barely amortized and custom runs even with, or slightly behind, a hosted platform. Push the same cluster toward the client's 200,000-call target and the fixed cost spreads across roughly ten times as many minutes, dropping the effective infrastructure cost per minute enough that the custom total falls meaningfully under the platform's flat $0.49. That crossover lands somewhere in the 10,000 to 20,000 calls-a-month range for us, below that a platform is the cheaper and faster choice, above it the fixed cost of owning the infrastructure starts paying for itself.
What we'd do differently on day one
Three things, in hindsight, would have saved real time.
We'd start with a streaming speech-to-text engine and never touch Whisper for anything beyond offline QA. The month we spent trying to force a batch model into a streaming pipeline is a month we didn't get back.
We'd constrain scope before we touched scaling logic. We eventually built airtight handling for the ten most common call outcomes, a hard fallback of hang-up-and-retry for everything else, and expanded that list incrementally against a regression dataset. Trying to handle five hundred scenarios at once early on meant we handled none of them well.
And we'd build the regression test before writing more call-handling logic, not after. Once we had a dataset of 1,100 real call recordings, split across roughly 400 phone-validation calls, 400 recall-routing calls, and 300 complex multi-scenario calls, and started running it before every deploy, our classification accuracy climbed from around 50 to 60 percent to about 80 percent in a matter of weeks. That dataset should have existed before the second week of the project, not the fourth month.
Where this leaves the number
Five cents a minute is a real number. It's just describing a smaller job than the one this system does, one without the IVR navigation, the classifier trained on our own data, or the structured CRM writes that make the rest of it hard. Ours costs more because more of the hard part happens inside our own infrastructure instead of somebody else's. Whether that trade is worth it depends entirely on whether your project needs the five things a platform can't do, and whether you're running enough volume for the fixed cost of owning it to pay for itself. For most people building a voice agent, it won't be. For a system that has to clear 200,000 calls a month against a client's live CRM, it was.
This system was built by the engineering team at Meduzzen, a Ukrainian software company that designs and ships production AI, voice, and backend systems (a longer, more detailed version of this write-up lives on the Meduzzen engineering blog). If you are staffing work like this, you can hire AI developers with real production experience, or hire developers in Ukraine across the wider stack.
References
[1] T. Stivers et al., "Universals and cultural variation in turn-taking in conversation," Proceedings of the National Academy of Sciences (2009).
[2] Coval, "Best STT Providers in 2026: Independent Benchmarks and How to Choose," coval.ai (2026).
[3] AIMultiple Research, "Speech-to-Text Benchmark: Deepgram vs. Whisper," aimultiple.com (2026).
[4] Salesforce AI Research, "VoiceAgentRAG: Solving the RAG Latency Bottleneck in Real-Time Voice Agents Using Dual-Agent Architectures," arXiv:2603.02206 (2026).
[5] Vonage Developer Blog, "Reducing RAG Pipeline Latency for Real-Time Voice Conversations," developer.vonage.com.
[6] OpenAI, "Latency Optimization," OpenAI API documentation, developers.openai.com; corroborated in "Measuring the Response Latency of OpenAI's WebRTC-Based Realtime API," webrtchacks.com (2026).
