Real-Time Voice AI: The Latency Budget

01The clock you are running against

Before any engineering, one measurement. It is the only number on this page you cannot negotiate with.

Stivers and colleagues timed the gap between a yes/no question and its answer in ten languages drawn from five continents. Every language produced a unimodal distribution with its mode between 0 and 200 ms. The cross-linguistic median was +100 ms. Language means ran from +7 ms in Japanese to +469 ms in Danish.

Your agent competes against that distribution, and 469 ms was the slow end of it. A turn that takes 859 ms sits at roughly eight times the cross-linguistic median and nearly double the slowest language mean measured.

Telephony arrived at a compatible number from the other direction. ITU-T Recommendation G.114 says that below 150 ms of one-way mouth-to-ear delay most applications feel almost transparent, and that above 400 ms the delay is unacceptable for general network planning.

So the working line comes from arithmetic and needs no benchmark. Two one-way trips at the ITU's unacceptable limit come to 800 ms, and the human distribution is almost entirely below 500 ms. Somewhere around 800 ms of turn latency, a reply stops landing anywhere a conversational gap has been observed to land.

02Where the milliseconds go

Nine stages, measured from the user's last phoneme to the first sample of the reply reaching their ear. Read the long bar first.

Each row, and where its number came from:

The budget rests on one assumption and eight sourced figures, and totals 859 ms. The shape matters more than the exact sum, so push on the numbers yourself.

import numpy as np

# One turn, end of the user's speech to first agent audio.
# Every row is sourced or flagged as an assumption on the page.
stage = ["capture + Opus encode", "network up", "jitter buffer",
         "endpointing", "ASR final transcript", "LLM first token",
         "TTS first audio", "network down", "playout buffer"]
ms = np.array([25, 30, 80, 295, 24, 220, 75, 30, 80], float)
total = ms.sum()

print(f"budget {total:.0f} ms")
for i in np.argsort(-ms)[:4]:
    print(f"  {stage[i]:<21} {ms[i]:5.0f} ms   {ms[i]/total:5.1%}")

moved = ms[[0, 1, 2, 7, 8]].sum()
print(f"\naudio moving, not thinking   {moved:.0f} ms")
print(f"LLM first token              {ms[5]:.0f} ms")

def edit(i, v):
    x = ms.copy(); x[i] = v; return x.sum()

print(f"\nhalve the LLM            {edit(5, 110):.0f} ms")
print(f"endpointer at 5% cutoff  {edit(3, 543):.0f} ms")
print(f"fixed 800 ms silence     {edit(3, 800):.0f} ms")

03Endpointing: deciding that you stopped

The largest single item in the budget is a decision and not a computation, and getting it right is the highest-leverage work in voice AI.

Voice activity detection is cheap. Silero VAD processes a 30 ms chunk in under a millisecond on one CPU thread, from a JIT model of about two megabytes. Knowing whether audio contains speech is a solved, nearly free problem.

Endpointing is the hard one: has this person finished, or are they thinking? A pause of 300 ms means both things, and the audio is identical.

Fixed thresholds, and why they cannot win

The default policy is a silence timer. Wait $T$ milliseconds of non-speech, then commit the turn. It has exactly two behaviours, and they are the two failure modes:

• False cut-off. A mid-sentence hesitation lasts longer than $T$, so the agent interrupts. The user has to start over, and the recovery costs several seconds.
• Dead air. The user finished, and you pay $T$ milliseconds of silence before anything happens. Every single turn pays it.

The trade is direct and it has no free parameter. Raising $T$ reduces cut-offs and raises dead air by exactly the same amount, because dead air is $T$.

Semantic endpointing moves the curve

The fix is to look at what was said as well as whether sound is present. "My account number is four two" is unfinished after 400 ms of silence. "That's all, thanks" is finished after 150 ms. A small language model reading the partial transcript can tell them apart.

LiveKit's Turn Detector v1, released in June 2026, distils a Qwen2.5-7B teacher into a 0.5B student for CPU inference, and reports numbers on an open benchmark called eot-bench:

Read that as a curve and not a ranking. The same post gives the inverse view: a 10% false-cutoff target costs 295 ms of mean latency, and a 5% target costs 543 ms. Halving your interruption rate costs 248 ms on every turn, which is the exchange rate you are buying.

import numpy as np
rng = np.random.default_rng(0)

N = 60000
is_end = rng.random(N) < 0.5          # the user really has finished
# Mid-turn hesitations in ms. A real turn end never resumes.
pause = np.where(is_end, np.inf,
                 rng.lognormal(np.log(240), 0.6, N))
hes = ~is_end

def fixed(T):
    """Wait T ms of silence, then commit. Dead air is always T."""
    return (hes & (pause > T)).sum() / hes.sum()

# A semantic endpointer reads the partial transcript at 120 ms and
# commits early when confident. The separation below is invented;
# only the shape of the curve is the point.
score = rng.normal(np.where(is_end, 1.7, 0.0), 1.0)

def semantic(c, T=800.0):
    early = score > c
    bad = (hes & early & (pause > 120)) | (hes & ~early & (pause > T))
    wait = np.where(early, 120.0, T)
    return bad.sum() / hes.sum(), wait[is_end].mean()

grid = np.arange(50, 3000, 5.0)
rates = np.array([fixed(t) for t in grid])

print(" false cut | fixed silence | semantic | saved")
for c in (0.4, 1.0, 1.6, 2.2):
    r, wait = semantic(c)
    t_match = grid[np.argmin(np.abs(rates - r))]   # same error rate
    print(f"   {r:6.2%}  |   {t_match:6.0f} ms   | {wait:5.0f} ms |"
          f" {t_match - wait:5.0f} ms")

04Barge-in, and the state it corrupts

Interruption is usually specified as "stop the audio". It is three separate problems, and the third one is where the bugs live.

Hearing the user through your own voice

The microphone picks up the loudspeaker. Without cancellation, the agent's own output is the loudest thing in the input, so the VAD fires on the agent and the agent interrupts itself.

Acoustic echo cancellation solves this by subtracting a filtered copy of the reference signal, which is why the playback path has to hand its samples to the AEC. RFC 7874 says WebRTC endpoints SHOULD include an AEC and deliberately mandates no particular algorithm.

The hard case is double-talk: both parties speaking at once, which is exactly what barge-in is. The adaptive filter must not diverge while it is being trained on a mixture. The ICASSP 2022 AEC Challenge scored entries on subjective quality across single-talk and double-talk conditions plus a speech recognition word acceptance rate, which is the right pairing: residual echo that a human tolerates can still wreck a transcript.

Audio you cannot take back

When you decide to stop, some of your speech has already left. Frames handed to the playout buffer are gone from your control. With the 80 ms buffer from section 02 plus a 20 ms frame, assume up to about 100 ms of speech reached the ear after your cancel.

This is a small number with a large consequence, because the agent said a word you believe it did not say.

Deciding it was an interruption at all

"Mhm", "right" and "yeah" are backchannels. They mean keep going, but a VAD classifies them as speech and stops the agent, which is why over-eager barge-in feels as broken as none at all.

Separating a backchannel from a real interruption needs the words, because energy alone does not separate them, so it lands on the same model that does semantic endpointing. In a cascade, this means your barge-in decision depends on your ASR having already produced a partial transcript for a 200 ms utterance.

Cancellation also has to reach every component. Stopping the synthesiser while the language model keeps generating leaves an orphaned response that arrives after the user's next question and answers the previous one.

05Streaming ASR: what a chunk costs

Three knobs decide the accuracy you pay for arriving early. Two of them turn out not to cost you turn latency at all.

A batch recogniser conditions on the whole utterance. A streaming one must commit to words while audio is still arriving, using everything before plus a short window after. That window has several names: lookahead, right context, emission delay, chunk latency.

NVIDIA published a clean version of the exchange rate for its cache-aware streaming Nemotron models in January 2026. Raising chunk latency from 0.16 s to 0.56 s drops word error rate from 7.84% to 7.22%.

Working it out, 400 ms of extra delay buys 0.62 WER points, about 0.16 points per 100 ms. The trade is unusually mild, which is the interesting part.

Why you cannot just chunk an offline model

Slicing audio into windows and running a batch model on each looks like streaming and is not. Pages 13 and 34 cover why the accuracy degrades. The cost that matters here is compute.

Sliding windows overlap, so the encoder repeatedly re-processes audio it has already seen, and recomputes attention over context it already had. A cache-aware streaming encoder keeps the activations instead and processes each frame once.

NVIDIA reports the size of that difference as concurrency: at a 320 ms chunk on an H100, 560 simultaneous streams against a 180-stream baseline, roughly three times. For a voice product, concurrency per GPU is the unit cost of a conversation, so this is a pricing decision wearing an architecture costume.

The third knob is the one nobody sets

Left context matters too. A streaming encoder that keeps unbounded history grows its state per stream, and memory per stream is what caps concurrency. Bounding it costs a little accuracy on long turns and is frequently the difference between one GPU and four.

06Streaming TTS: only the first chunk matters

Total synthesis time is almost irrelevant. Time to the first audible sample is the entire metric, and the biggest term in it is not in the synthesiser.

Once playback starts, generation only has to keep up with real time. A real-time factor below 1.0 means the buffer never drains. Everything else is time to first audio, and it decomposes further than most teams look.

ElevenLabs' own latency documentation is unusually candid about this. Flash model inference is around 75 ms for short inputs. Network round trips are 20 to 200 ms. A 500 ms player buffer is common. The model is the small term.

The chunking decision that costs half a second

The synthesiser cannot start until it has text, and the text arrives token by token. What you choose to wait for is therefore a latency decision made upstream of the TTS.

Take the arithmetic: at 40 tokens per second, one token is 25 ms. Waiting for a complete 20-token sentence costs 500 ms after time to first token. Cutting at the first clause boundary, say 6 tokens, costs 150 ms.

This is a 350 ms difference from a text-splitting rule, larger than anything you will win by changing synthesiser. It is also why "we stream the TTS" is an ambiguous claim worth interrogating: streaming audio out while buffering text in to a sentence boundary gives you the latency of a non-streaming system.

Where the vocoder sits in this

The last stage turns frames into samples, and it cannot start from a single frame. Convolutional vocoders need enough frames to fill their receptive field, plus some lookahead, before the first sample exists. Chunks are also spliced with an overlap from the previous chunk so the seam is not audible.

Whether that stage is your bottleneck depends entirely on where you run. A 2025 comparison of low-latency neural vocoders found that on CPU-only systems the per-chunk streaming overhead, data movement and parameter loading, dominates raw arithmetic at low latencies. The same work reports that a single frame of lookahead recovers nearly all of the quality of a non-causal model.

On a server GPU the picture inverts: the autoregressive token generation in front of the vocoder is the slow part, and the vocoder is a rounding error. Claiming one universal bottleneck is how teams end up optimising the wrong stage.

07Cascade or speech-to-speech

The 2026 trade is real and it is not close to settled. Both options give up something specific, and the specifics are what decide it.

A cascade is ASR, then a text model, then synthesis. A native speech model consumes and emits audio, with no text bottleneck in the loop. Page 34 has the model-by-model view; this is the engineering consequence.

Be precise about what the cascade loses, because "information loss" undersells it. The transcript destroys hesitation, sarcasm, emphasis and emotional state, and it also destroys timing. Overlapped speech becomes either one sequence or two, and which one is a decision your diarizer made.

The cascade also serialises three separate warm-up latencies, and gives you three services that can independently rate-limit, retry or go down mid-turn.

What speech-to-speech loses is mostly observability. You cannot log what you never rendered, so transcripts, retrieval keys, quality evals, compliance review and text guardrails all need a parallel ASR pass that you were trying to avoid running.

08Transport: why WebRTC, and what loss does

245 ms of the budget is audio in transit or sitting in a buffer. The protocol choice underneath that is not incidental.

Why not HTTP

TCP guarantees ordered delivery, so a lost segment is retransmitted and every later segment waits behind it. This is head-of-line blocking, and it is the wrong guarantee for a live conversation.

Audio that arrives 200 ms late is worthless; its moment has passed. It is better to conceal 20 ms of missing sound than to stall the next 200 ms waiting for it. WebRTC carries audio over SRTP on UDP, so a lost packet leaves a hole and does not stall the stream.

WebSockets over TCP remain fine for the control plane and for server-to-server hops on a reliable network. They are the wrong choice for the last mile to a phone on a train.

Opus, and the 25 ms you pay before sending anything

RFC 7874 makes Opus and G.711 mandatory for WebRTC endpoints, and requires decoders to support all Opus modes. Opus is not a preference here, it is the floor.

RFC 6716 gives its structure. The SILK layer handles 10 to 60 ms frames and needs 5 ms of extra look-ahead for noise shaping. The CELT layer handles 2.5 to 20 ms frames with 2.5 ms of overlap look-ahead. A 20 ms frame plus SILK's 5 ms is the 25 ms in the budget table, spent before a single packet is on the wire.

Opus also carries in-band forward error correction: a low-bitrate copy of the previous frame rides inside the next packet, so one isolated loss is recovered without a retransmission and without a round trip. It covers exactly one frame back, so a burst of four defeats it. Mozilla's 2016 experiments found the effect subtle at low loss rates and very noticeable at high ones.

The jitter buffer is a latency knob

Packets sent every 20 ms do not arrive every 20 ms. A buffer absorbs the spread by holding audio back, which is why it appears twice in the budget.

WebRTC's NetEq pulls 10 ms of audio at a time and adapts continuously. Its delay manager keeps a histogram of packet inter-arrival times in 20 ms buckets and targets roughly the 97th percentile, starting from 80 ms. When the buffer runs long or short it time-stretches the audio, accelerating or expanding, and it does not drop or stall.

import numpy as np
rng = np.random.default_rng(1)

# 20 ms Opus packets. One-way delay is a 30 ms floor plus a heavy
# right tail: most packets are early, a few are very late.
n = 400000
sent = np.arange(n) * 20.0
jitter = rng.gamma(2.0, 7.0, n)          # ms
arrive = sent + 30.0 + jitter

print("buffer   concealed   added delay")
for depth in (0, 20, 40, 60, 80, 120):
    deadline = sent + 30.0 + depth
    late = (arrive > deadline).mean()
    print(f"{depth:4} ms   {late:8.3%}   {depth:8} ms")

q97 = np.quantile(jitter, 0.97)
print(f"\n97th percentile of the jitter: {q97:.0f} ms")
print("NetEq sets its target near that quantile, then adapts.")

What concealment does to the recogniser

A concealed packet is not silence. The decoder invents plausible audio to bridge the gap, tuned to sound acceptable to a human ear. Your ASR was not trained on it.

A 2022 study over G.722 telephony found the behaviour is a knee and not a slope: a clean-trained recogniser held up to about 10% packet loss and then degraded roughly in proportion, while a model trained on network-distorted speech pushed that knee out to about 15%. Training on the distortion you will see moves the cliff.

There is a cheaper version of the same idea. Dissen and colleagues showed at Interspeech 2024 that a front-end adaptation network in front of a frozen Whisper, trained against the recogniser's own loss, recovers much of the loss-induced error without retraining the ASR.

09Build this

The budget in section 02 is a worked example with one assumed row. Yours will look different, and the difference is the only thing worth acting on.

10What breaks

Almost none of these are model failures, and almost all of them show up as "it feels laggy".

• p50 is fine and p95 is three seconds. One slow hop, a cold start or a retry. Users remember the tail, and a mean hides it completely.
• The endpointer was tuned in an office. Staff speak in fluent complete sentences to a demo. Customers hesitate, read numbers aloud and pause to find documents.
• Barge-in stops the audio but not the model. The orphaned generation arrives later and answers the previous question.
• The transcript records what was generated and not what was heard. After every interruption the model believes it delivered a sentence that was cut off mid-word.
• The synthesiser buffers to a sentence boundary. Streaming audio out while blocking on text in gives you non-streaming latency behind a streaming API.
• The jitter buffer is a constant. Tuned on a wired desk connection, then deployed to mobile networks, where it is now too shallow.
• Echo cancellation fails on a speakerphone. The agent hears itself, the VAD fires, and it interrupts itself in a loop.
• The first tool call doubles the turn. Nothing streams across a function call, so the user gets the full budget twice with no audio in between.
• Backchannels stop the agent. Someone says "mhm" and the agent goes quiet, which reads as being ignored.
• WER was measured on clean files. The production path adds a codec, concealment and possibly an 8 kHz transcode that no benchmark includes.

11Where you meet this in the wild

Same budget, very different constraints on which lines you can touch.

12Interview questions

13Go deeper

●Now write it yourself

Reading the derivation and being able to produce it are different skills. These are Deep-ML problems that exercise what this page covers — each one is checked against real test cases, not multiple choice.

Matched to this page from Deep-ML's catalogue of 1,380 problems. More at deep-ml.com, and Where to practise covers the other platforms and what each one trains.

14Related pages