Case study
A real-time voice assistant for sales conversations, rebuilt from first principles once I measured where the seconds and the dollars actually went.
Every figure below is measured or derived from published pricing — nothing estimated without saying so.
Story one: the pipeline, not the model, was the product problem.
A single spoken reply ran through six sequential steps with zero overlap: a fixed 1.5-second wait to confirm the user had stopped talking, an upload, a transcription, a complete LLM generation, a complete audio synthesis, then a download. Nothing started until the previous step finished, so the user heard silence for the sum of all of it. Five of the six were network round-trips; the silence wait was local turn detection.
The cost split was equally counter-intuitive. Of an estimated ~$0.0069 per reply, the voice layer accounted for roughly 87% and the language model about 10%. Self-hosting therefore breaks even at about 85,000 voice replies per month, but not until roughly 730,000 replies for the LLM — so the voice layer is the only part worth bringing in-house early. Both shares are estimates from published pricing.
That wait exists to be sure the customer has actually finished speaking. Streaming does nothing for it. It shrinks with better turn detection — streaming voice activity detection (VAD) that decides on end-of-turn as the audio arrives instead of waiting out a fixed timer.
Once the turn is detected, sentence-level streaming is what removes the dead air: transcription, generation and synthesis overlap instead of each waiting for the last to finish completely.
Trade-off and lesson: measure the whole pipeline before optimizing any single component. The most-discussed piece — the language model — was neither the slowest nor the most expensive part of the product.
Story two: the shipped app called hosted APIs — this profiling was done on a self-hosted model to understand the physics and test whether in-housing was viable.
Processing a 512-word prompt took 0.76s and pushed GPU compute to about 47% — the model reads the whole input in parallel, so raw arithmetic is the limit. That is slow for a short prompt, which tells you the run was a large model on a modest single GPU rather than a tuned production deployment.
Generating each token took 15.5ms at about 82% memory bandwidth and only 5% compute — every single token reloads all model weights from memory.
By turn 10, latency had degraded 7× — 661ms to 4,649ms — because every turn re-reads the whole conversation transcript before it can answer. Capping how much history is carried forward held it at roughly 1.5–1.7s.
Every turn, the model re-reads the whole conversation so far. Left unbounded, turn 10 meant thousands of accumulated words to process before the first sound — 4.6 seconds of dead air on a live call. Capping what gets carried forward fixes the ceiling. The layout below is the trimming policy I designed for this, not a measured production configuration.
3,500+ words re-read per reply.
4,649 ms
Fixed budget, roughly 800 words per reply.
~1,500 ms
Instructions, persona and call objective are never trimmed.
Older turns collapse into a short rolling summary instead of raw transcript.
The last couple of turns stay verbatim so recent specifics survive.
Modeled, the same read step took 8.5s on an L4, 1.3s on an A100 and 0.8s on an H100. The cheapest GPU per hour lacked the memory bandwidth voice needs, which made it both too slow for real-time conversation and more expensive per usable concurrent reply.
A separate axis from the GPU: how much CPU and RAM the serving container gets — 4, 8 or 16 vCPU. This one barely touches token generation speed. What it moves is how fast a cold container comes up and is ready to take a call, how many concurrent sessions one instance can hold, and the idle bill between calls. Under-provision and the first caller of a quiet period waits through a cold start; over-provision and you pay for capacity that sits idle.
The two have to be sized independently: GPU by memory bandwidth for speed, vCPU and RAM by cold-start tolerance and concurrency for cost.
Trade-off and lesson: for conversational workloads, select hardware by memory bandwidth — not by peak compute numbers or hourly price.
Story three: pick any two — until you change the shape of the problem.
A local open-source voice engine cost only compute but sounded robotic and took real effort to shape into a persona. A hosted API gave a natural voice and fast integration at a per-character price. The highest-fidelity option sounded best and added delay the user could feel. No single engine won on all three dimensions.
The resolution was architectural rather than a choice between engines: chunk the model output at sentence boundaries and stream each sentence to the voice engine in parallel, so synthesis of the next sentence happens while the current one is already playing. That masks most of the latency and brings time-to-first-audio down to the edge of human perception.
Trade-off and lesson: streaming changes the trade-off instead of resolving it — you stop choosing a side of the trilemma and start hiding the cost of the side you kept.
Key learning: real-time AI products are systems problems. The decisions that mattered — pipelining, context trimming, hardware selection, sentence chunking — were all made outside the model.