Case study
How long a caller waits before a voice agent starts talking, and what it costs — a full speech-to-text, LLM and text-to-speech pipeline on one self-hosted AMD MI300X, NVIDIA GH200 and NVIDIA H100 PCIe, measured against managed APIs.
Measured, not modeled. APIs run 2026-10-01, MI300X and TTS 2026-10-03, H100 PCIe and GH200 2026-10-04. Replies capped at 150 tokens; SLO is TTFT p95 under 500 ms.
Turning the modeled hardware story into measured numbers.
The Voice AI case study ended on three claims: token generation is memory-bandwidth bound, GPU choice drives reply speed (modeled at 8.5 s on an L4 versus 0.8 s on an H100), and self-hosting only pays off once call volume is steady. All three were derived from profiling and published pricing. None had been put on real data-center hardware against the APIs a team would actually buy.
This benchmark does that. The question a product owner needs answered is not "how many tokens per second" but "how long does a caller sit in silence, how many callers can one GPU hold before that silence breaks the SLO, and what does each answer cost." So the metric is speech in → first audio out, the SLO is 500 ms at p95, and every platform runs the same 12 realistic phone-call turns with the same scripts.
Voice turn = speech-to-text + LLM time to first sentence. Sessions within SLO = concurrent callers at TTFT p95 under 500 ms.
| Platform | TTFT p50 | TTFT p95 | Per-user tok/s | Peak tok/s (8 users) | ASR p50 | Voice turn | $ / 1M out | Sessions in SLO |
|---|---|---|---|---|---|---|---|---|
AMD MI300X Llama-3.1-8B · self-hosted | 14 ms | 15 ms | 195 | 1,051 | 111 ms | 242 ms | $0.079 | 256+ sessions |
NVIDIA GH200 Llama-3.1-8B · self-hosted | 17 ms | 18 ms | 175 | 1,289 | 71 ms | 224 ms | $0.077 | 128 sessions |
NVIDIA H100 PCIe Llama-3.1-8B · self-hosted | 41 ms | 43 ms | 96 | 671 | 172 ms | ~434 ms | $0.198 | 64 sessions |
Azure AI Foundry gpt-4.1-mini | 784 ms | 1,018 ms | 213 | 203 | 638 ms | 1,517 ms | $2.08 | 0 sessions |
OpenAI gpt-4.1-mini | 454 ms | 574 ms | 97 | 379 | 1,079 ms | 1,740 ms | $2.08 | 0 sessions |
OpenAI gpt-4o-mini · reference | 518 ms | — | 95 | 297 | 1,766 ms | 2,495 ms | $0.794 | 0 sessions |
Self-hosted rows run Llama-3.1-8B with the client on the same machine, so they carry no network time; their cost is taken at the highest load where every level up to it met the SLO (MI300X 256 callers, GH200 128, H100 PCIe 64). gpt-4o-mini is from an earlier OpenAI run, kept for reference. The same Llama model can't be compared as a managed service: Azure AI Foundry retired Meta-Llama-3.1-8B-Instruct on 2026-06-13.
One caller. Where each pipeline spends the silence.
Self-hosted rows are one measured pipeline each (p50, 1 caller); the bar shows stage p50s and the label the measured end-to-end p50. API rows add stages measured separately: ASR + LLM from the October 1 runs, TTS from Azure AI Speech (Ava neural) and gpt-4o-mini-tts on October 3.
What changed my mind: in the original voice AI work the voice layer was the expensive part. Here the managed voice layer is actually fast — Azure's TTS is one of the quickest stages on the chart. The managed penalty comes from speech-to-text and the LLM round-trip, which is exactly the part a co-located GPU removes.
Published vendor specs; rental prices as seen in October 2026.
| Spec | NVIDIA H100 PCIe tested | NVIDIA H100 SXM not yet tested | NVIDIA GH200 tested | AMD MI300X tested |
|---|---|---|---|---|
| Memory size | 80 GB | 80 GB | 96 GB (+480 GB CPU memory) | 192 GB |
| Memory type and bandwidth | HBM2e, ~2.0 TB/s | HBM3, ~3.35 TB/s | HBM3, ~4 TB/s | HBM3, ~5.3 TB/s |
| Power limit | 350 W | 700 W | up to ~1,000 W (GPU + CPU module) | 750 W |
| Rental price used | ~$2.49/hr (Lambda) | ~$2.99–3.29/hr | $1.99/hr (Lambda), ARM CPU | $1.99/hr (AMD Developer Cloud) |
Token generation for an 8B model is limited mostly by memory bandwidth, so the PCIe H100 is the mid-tier card and the SXM H100 is the like-for-like rival to the MI300X. The GH200 pairs an H100-class GPU (with faster HBM3) with a 72-core ARM Grace CPU on one module. Cost per token is normalised by each GPU's own hourly price, so it is a fair comparison either way.
LLM only, whole GPU: Llama-3.1-8B-Instruct swept from 1 to 256 concurrent calls.
| Metric | AMD MI300X | NVIDIA GH200 | NVIDIA H100 PCIe |
|---|---|---|---|
| Rental | $1.99/hr | $1.99/hr | $2.49/hr |
| TTFT p50 / ITL p50, 1 caller | 14 ms / 4.9 ms | 17 ms / 5.4 ms | 41 ms / 9.4 ms |
| Highest consistent load within 500 ms p95 | 256 callers | 128 callers (256 failed 3 of 3) | 64 callers |
| Output tok/s at that load | 7,026 | 7,140 | 3,487 |
| $ per 1M output tokens | $0.079 | $0.077 | $0.198 |
| Full voice loop, first audio, 1 caller | 463 ms | 331 ms | 641 ms |
| Burst, 512 in / 128 out, total tok/s | 18,071 | 30,033 | 18,591 |
| vLLM | 0.27.1 (ROCm 7.2) | 0.31.0 (CUDA, ARM) | 0.30.0 (CUDA) |
Against the H100 PCIe the MI300X wins clearly: about 2× the throughput inside the SLO at about 40% of the cost per token. Against the price-matched GH200 it is close: a tie on cost per token, behind on end-to-end latency and on prompt-heavy bursts, and ahead on how many calls one GPU holds inside the SLO (256 vs 128), thanks to 192 GB of memory. GH200 high-load levels ran three times; MI300X and H100 PCIe once. Newer vLLM on the NVIDIA runs may account for part of their lead.
concurrent callers
Red bars miss the SLO. The MI300X stays under 500 ms at 256 calls (418 ms); the GH200 holds to 128 and fails at 256 (748–809 ms); the H100 PCIe crosses it at 128 (548 ms).
| Calls | TTFT p95 | Output tok/s | ||||
|---|---|---|---|---|---|---|
| MI300X | GH200 | H100 | MI300X | GH200 | H100 | |
| 1 | 15 ms | 18 ms | 43 ms | 195 | 175 | 96 |
| 4 | 22 ms | 23 ms | 62 ms | 561 | 674 | 355 |
| 8 | 25 ms | 26 ms | 72 ms | 1,051 | 1,289 | 671 |
| 16 | 32 ms | 39 ms | 122 ms | 1,979 | 2,388 | 1,256 |
| 32 | 54 ms | 73 ms | 200 ms | 2,664 | 3,932 | 2,181 |
| 64 | 92 ms | 95 ms | 266 ms | 3,621 | 6,178 | 3,487 |
| 128 | 222 ms | 241 ms | 548 ms | 6,518 | 7,140 | 4,201 |
| 256 | 418 ms | 748–809 ms | 438 ms | 7,026 | ~7,450 | 4,449 |
GH200: 64 and 128 callers are the median of 3 runs (pass 3/3); 256 failed all 3 runs (p95 748–809 ms). H100 PCIe: missed the SLO at 128 (548 ms) and passed at 256 (438 ms) on a single run each, so its consistent limit is 64.
An 8B model in 16-bit is about 16 GB of weights, and every token reads all of it. The bandwidth floor is roughly 3 ms per token on the MI300X (~5.3 TB/s), 4 ms on the GH200 (~4 TB/s) and 8 ms on the H100 PCIe (~2.0 TB/s). Measured single-caller decode: 4.9, 5.4 and 9.4 ms. Same order, each close to its floor — the memory-bound behaviour the earlier profiling found at 15.5 ms per token on a much smaller GPU.
On the MI300X, going from 1 to 256 callers raises per-token latency 3.3× (4.9 → 16.2 ms) but output throughput 36× (195 → 7,026 tok/s). Each extra caller rides on weights the GPU was already reading. The prompt-heavy burst test is compute-bound instead, and there the GH200 leads clearly (30,033 tok/s vs ~18,000 for the other two).
Whisper → Llama → Kokoro on one GPU, same models, memory split and single TTS worker on all three. The LLM was never the problem.
| Calls | STT | LLM | TTS | Audio | p95 |
|---|---|---|---|---|---|
| 1 | 71 | 14 | 107 | 331 | 412 |
| 4 | 149 | 24 | 209 | 589 | 777 |
| 8 | 160 | 24 | 726 | 1,158 | 1,345 |
| 16 | 187 | 26 | 1,814 | 2,207 | 2,494 |
| 32 | 365 | 30 | 3,365 | 3,847 | 4,829 |
All times in ms, p50 except the last column.
Fastest stack measured: speech-to-text (71 ms) and text-to-speech (107 ms) are both quicker than on the MI300X (111 / 221 ms); the Grace CPU beside the GPU likely helps TTS text pre-processing.
| Calls | STT | LLM | TTS | Audio | p95 |
|---|---|---|---|---|---|
| 1 | 111 | 12 | 221 | 463 | 598 |
| 4 | 123 | 14 | 626 | 923 | 1,008 |
| 8 | 118 | 13 | 1,588 | 1,884 | 2,160 |
| 16 | 124 | 14 | 3,414 | 3,727 | 3,958 |
| 32 | 401 | 19 | 6,653 | 7,066 | 7,670 |
All times in ms, p50 except the last column.
STT and the LLM barely move with load; the single TTS worker is the queue.
| Calls | STT | LLM | TTS | Audio | p95 |
|---|---|---|---|---|---|
| 1 | 172 | 43 | 220 | 641 | 844 |
| 4 | 160 | 50 | 421 | 995 | 1,288 |
| 8 | 160 | 51 | 1,408 | 1,965 | 2,122 |
| 16 | 178 | 51 | 3,313 | 3,901 | 4,218 |
| 32 | 362 | 59 | 7,003 | 7,625 | 8,625 |
All times in ms, p50 except the last column.
TTS matches the MI300X (~220 ms); the gap is speech-to-text (172 vs 111 ms) and the LLM (43 vs 12 ms first token).
Whisper-large-v3-turbo (20% of GPU memory) → Llama-3.1-8B (60%) → Kokoro-82M, one TTS worker. On every GPU, STT and the LLM barely move with load while TTS time climbs with each caller: the single TTS worker is the queue.
Trade-off and lesson: a GPU benchmark that only measures the LLM would have said "128–256 callers per GPU." The full loop says first audio passes 500 ms by 4 callers with one TTS worker — on every GPU tested. Capacity planning has to follow the slowest stage, and here that is an 82M-parameter voice model, not the 8B LLM.
8 voice callers alongside 32 long batch jobs (~1,500 words in, 512 out).
| Phase | Voice TTFT p50 | Voice TTFT p95 | vs alone | Batch out tok/s | Cost split voice / batch |
|---|---|---|---|---|---|
| H100 PCIe | |||||
| Voice alone | 53 ms | 89 ms | 1.0× | — | — |
| With batch neighbour | 91 ms | 131 ms | 1.5× | 2,824 | $0.00019 / $0.00441 |
| Neighbour, voice at higher priority | 90 ms | 132 ms | 1.5× | 2,826 | $0.00018 / $0.00430 |
| GH200 | |||||
| Voice alone | 18 ms | 26 ms | 1.0× | — | — |
| With batch neighbour | 24 ms | 40 ms | 1.6× | 5,034 | $0.00005 / $0.00178 |
| Neighbour, voice at higher priority | 26 ms | 33 ms | 1.3× | 5,057 | $0.00005 / $0.00171 |
A batch neighbour made voice 1.5–1.6× slower but still inside the SLO. On the H100 PCIe, vLLM request priority changed nothing: it only reorders a queue, and with free KV-cache memory nothing queued. On the GH200 it trimmed voice p95 from 40 to 33 ms. Cost is split by processed tokens; the batch tenant used ~96% of them.
Trade-off and lesson: a priority flag is not isolation. If voice has to share a GPU with batch work, the guarantee has to come from separate capacity — MIG partitions, a separate replica, or a cap on the batch tenant.
APIs at list price. GPUs at $/hr ÷ throughput at their consistent SLO load — a floor, since real utilisation is lower.
| Provider | Callers | p50 | p95 | Sentence | Errors |
|---|---|---|---|---|---|
| Azure AI Speech | 1 | 111 ms | 296 ms | 189 ms | 0/32 |
| Azure AI Speech | 4 | 76 ms | 232 ms | 159 ms | 10/32 · 429 |
| Azure AI Speech | 8 | 51 ms | 243 ms | 129 ms | 16/32 · 429 |
| OpenAI gpt-4o-mini-tts | 1 | 591 ms | 1,508 ms | 1,387 ms | 0/32 |
| OpenAI gpt-4o-mini-tts | 4 | 478 ms | 818 ms | 1,173 ms | 0/32 |
| OpenAI gpt-4o-mini-tts | 8 | 408 ms | 600 ms | 1,046 ms | 0/32 |
From Azure Cloud Shell, 32 requests per level. Azure ran on the free F0 tier, which rate-limits concurrent requests (HTTP 429), so only its 1-caller row is clean.
More calls per GPU on memory-bound work, a tie on cost with the GH200, paid for in engineering time.
Generating each token reads the whole model from memory, so decode speed tracks memory bandwidth: ~5.3 TB/s on the MI300X, ~4 TB/s on the GH200, ~3.35 TB/s on the H100 SXM and ~2.0 TB/s on the H100 PCIe. That shows up as lower TTFT and ITL.
Memory left after the weights holds each live conversation's KV cache. More room means more callers batched at once before anyone queues: 256 callers inside the SLO on the MI300X, against 128 on the 96 GB GH200.
AMD Developer Cloud ($1.99/hr) is AMD's own developer programme with DigitalOcean, likely priced to win developers. Lambda prices the GH200 at the same $1.99/hr, which is why the two tie on cost per token.
| Trade-off | AMD MI300X (ROCm) | NVIDIA (CUDA) |
|---|---|---|
| Software maturity | Mainstream stacks (vLLM, PyTorch) work, with rough edges | Default target for nearly all AI software |
| Issues hit in this benchmark | MIOpen re-tuning (5.5 s per sentence until disabled), missing torchcodec, 2 of 8 TTS processes silently on CPU | One missing Python package (pyarrow) |
| Optimised tooling | Fewer tuned kernels; no TensorRT-LLM | TensorRT-LLM, mature FP8, tuned attention kernels |
| Availability | Smaller cloud footprint; account GPU limits | Offered by almost every cloud |
| Sharing one GPU | Partition modes exist, less used | MIG mature, well supported in Kubernetes |
| Scaling across GPUs | Infinity Fabric | NVLink / NVSwitch, more proven at large scale |
| Engineering time | More time debugging the stack | Mostly works out of the box |
Limits of this result: the H100 tested was the bandwidth-limited PCIe card; an 8B model is the MI300X's easiest case, and its 192 GB matters most for 70B+ models that need two H100s; prompt-heavy (prefill) work is compute-bound and favoured the GH200; vLLM versions differed (0.27.1 vs 0.30.0 and 0.31.0). In one line: the MI300X wins on memory, so it holds the most concurrent calls per GPU and beats the H100 PCIe on tokens per dollar; NVIDIA's price-matched GH200 ties it on cost and is faster end to end; and the MI300X's lower price comes with a less mature stack, paid for in engineering time.
One GH200 gets a caller to first audio in 331 ms, an MI300X in 463 ms, an H100 PCIe in 641 ms. Azure AI Foundry + Azure AI Speech take about 1.6 s and OpenAI about 2.3 s — before the self-hosted network hop is even added.
Every managed run misses it at p95, even with a single caller. With the LLM alone, the MI300X holds it up to 256 concurrent calls, the GH200 up to 128 and the H100 PCIe up to 64.
Clearly ahead of the H100 PCIe (2× throughput, 40% of the cost per token). Even with the price-matched GH200 on cost, behind it on latency and prompt-heavy bursts, ahead on calls per GPU.
Azure AI Speech starts audio in 111 ms — faster than the single Kokoro worker on the MI300X or H100 (~220 ms), and level with the GH200 (107 ms). The self-hosted lead comes from speech-to-text and the LLM.
Per 1M output tokens at full load, on either the MI300X or the GH200. Idle GPU time raises the self-hosted figure, so the break-even depends on steady call volume — the same utilisation question the original voice AI economics turned on.
A batch neighbour slowed voice 1.5–1.6× on both NVIDIA GPUs, still inside the SLO. vLLM request priority did nothing on the H100 PCIe and only trimmed GH200 p95 from 40 to 33 ms. Real isolation needs separate capacity.
Same workload everywhere, failures counted, limits stated up front.
12 realistic phone-call turns (booking, billing, sales objections), replies capped at 150 tokens (~20 s of speech), 3 warm-up requests, failures counted. Audio from LibriSpeech clips.
Azure AI Foundry gpt-4.1-mini and OpenAI gpt-4.1-mini (Oct 1); Azure AI Speech (Ava neural, F0 tier) and OpenAI gpt-4o-mini-tts (Oct 3). All called from Azure Cloud Shell, so network time is included.
AMD Developer Cloud (MI300X) and Lambda (H100 PCIe, GH200), one GPU each, with all three models sharing that single GPU and the same scripts on every machine.
Limits of this comparison: different LLMs (8B open model vs. gpt-4.1-mini); self-hosted figures exclude network time; API voice turns sum separately measured stages; Azure TTS above 1 caller was rate-limited; repeats only on the GH200's high-load levels. Scripts and raw results are public at globaledgepathway/voice-inference-benchmark.
Added here as they complete.
run_h100.sh, ~1 hrsame software as the GH200same footing as the API rowsreal calls-per-GPU figureKey learning: for voice, the benchmark that matters is time to first audio at p95 under concurrent load, measured through every stage. There is no single winning GPU: memory bandwidth and capacity decide how many callers one GPU holds, the rest of the stack decides how fast each one hears a reply, and whichever stage queues first sets the real limit.