Preloader
Others
  • Estimated reading time: 4 Minutes

Building a Real-Time Voice AI Pipeline: STT, LLM, TTS

Building a Real-Time Voice AI Pipeline: STT, LLM, TTS

Voice AI agents have moved from demos to production. They answer support lines, book appointments and call sales leads. If you have ever tried to build one, though, you know the gap between a demo that works in a quiet room and a system that handles real phone calls is large.

This article walks through the architecture of a real-time voice agent, the latency budget that decides whether it feels natural, and the practical problems that show up once real people start talking to it.

The basic pipeline

Most production voice agents use a cascaded pipeline with three main stages:

  1. Speech-to-text (STT): converts the caller's audio into text as they speak.
  2. Language model (LLM): decides what to say next, using the transcript, the conversation history and any business logic or tools.
  3. Text-to-speech (TTS): turns the reply into audio and streams it back to the caller.

Around that core sit the telephony layer (SIP or a cloud phone provider), a voice activity detector, an orchestration service that manages turns, and integrations with external systems such as calendars and CRMs.

Speech-to-speech models, which skip the text step, are improving quickly. Many teams still prefer the cascaded design because each stage can be tested, swapped and logged on its own, and the text transcript is useful for summaries and compliance.

The latency budget

Humans expect a reply within about a second of finishing a sentence. Much longer and the call feels robotic; callers start repeating themselves or hang up. That gives you a tight budget to share between every stage:

Stage

What eats time

End-of-speech detection

Waiting to be sure the caller has finished

STT finalisation

Converting the last words to text

LLM first token

Prompt processing and the first piece of the reply

TTS first audio

Generating the first chunk of speech

Network and telephony

Moving audio between services and the phone network

The key technique is streaming everywhere. Stream audio into STT as it arrives. Start the LLM as soon as you have a final transcript. Send the LLM's first sentence to TTS before the rest is generated, and start playing audio as soon as the first chunk is ready. A short, natural opener such as "Sure, let me check that" buys time while the rest is produced.

Co-locating services in the same region and keeping prompts compact also make a noticeable difference.

Turn-taking and interruptions

Real callers do not wait politely. They interrupt, talk over the agent, pause mid-sentence and answer questions before they are asked. Handling this well is what separates a usable agent from a frustrating one.

  • Endpointing: decide when the caller has finished. Pure silence thresholds are too slow or too eager; combining voice activity detection with the content of the partial transcript works better.
  • Barge-in: when the caller starts speaking, stop TTS playback immediately and discard the rest of the planned reply.
  • State tracking: keep a structured record of what has been learned, so the agent skips questions the caller has already answered.
  • Backchannel tolerance: ignore "mm-hmm" and "yeah" so they do not trigger a full interruption.

Tools and business logic

A voice agent that only talks is rarely useful. Production agents call tools: checking calendar availability, booking an appointment, looking up a record or writing notes to a CRM.

Two lessons apply here. First, tool calls add latency, so run them in parallel with speech where possible, or say something natural while waiting. Second, keep hard rules out of the prompt. Things like calling hours, opt-outs and escalation to a human should be enforced in code, not left to the model.

A real-world example: real estate lead calls

Real estate is a demanding test case. New leads arrive from portals and websites at all hours, and the first agent to call often wins the client. Products like Mergro use this kind of pipeline to call a new lead within a minute, ask a few qualifying questions, book a viewing and log the conversation in the team's CRM.

The architecture choices show up directly in the product. Some platforms give teams a configurable call-center toolkit to build their own agents, while others ship a narrower, ready-made agent for one job. This Mergro vs Structurely comparison is a good illustration of that build-versus-buy trade-off in one industry.

Testing and observability

Voice agents fail in ways unit tests do not catch. Useful practices include:

  • Recording and transcribing every call, with consent, for review
  • Tracking latency per stage, not just end to end
  • Running scripted test calls with background noise, accents and interruptions
  • Flagging calls where the caller hung up early or asked for a human
  • Reviewing a sample of real calls every week

Takeaway

The language model is only one part of a voice agent. Latency, turn-taking, tool design and observability decide whether callers stay on the line. Build for streaming from day one, enforce business rules in code, and test with messy, real-world audio. Do that, and a voice agent can handle conversations that used to need a person on the phone.

Related articles
Weekly trending
Do Cheap SEO Services Really Work? The Truth About Affordable SEO
18 Nov, 2026
  • Estimated reading time: 13 Minutes
Build-or-Buy Gets Harder When Code Gets Cheap
18 Nov, 2026
  • Estimated reading time: 4 Minutes
New Zealand Proxy Servers: How They Work and Why They Matter
18 Nov, 2026
  • Estimated reading time: 8 Minutes
Our Sponsors

Our blog is proudly supported by industry-leading sponsors.