If you're running LLMs in production and your serving setup is one pool of GPUs doing both prompt processing and token generation, you're leaving significant throughput on the table. Not in a theoretical, "someday we'll optimize" way. Right now, in mid-2026, every major serving framework supports the fix and teams like Baseten are reporting 50% lower time-to-first-token and 61% more requests per second after switching.
The pattern is called disaggregated prefill/decode serving. It's been production-ready since NVIDIA Dynamo 1.0 went GA in March, and it's now supported natively by vLLM, SGLang, TensorRT-LLM, and LMDeploy. If you haven't looked at it yet, this is the week.
Why Prefill and Decode Don't Belong on the Same GPU
LLM inference has two fundamentally different phases.
Prefill is where the model processes your input prompt. If you send a 50,000-token context, the model has to attend over all of it to generate the first token. This is compute-bound: you want as much raw FLOPS as possible. Your GPU's tensor cores are maxed out.
Decode is what comes after: generating one token at a time, auto-regressively, until you hit the stop condition. Each step reads the entire model weight set and the growing KV cache from GPU memory. Decode is memory-bandwidth-bound. You don't need more compute. You need faster memory access.
The problem is that a GPU optimized for one isn't ideal for the other. When both phases run on the same GPU pool, they compete. Long prefill spikes interrupt decode batches and spike TTFT. Decode batches fill up GPU memory and slow down prefill throughput. The GPU is always doing something, but it's rarely doing either thing optimally.
Disaggregated serving routes each phase to a dedicated worker pool. Prefill workers process incoming prompts at full compute utilization without waiting for decode batches to drain. Once prefill finishes, the KV cache (the model's "memory" of the prompt) is transferred over the network to a decode worker via a protocol called NIXL, which is the standard transfer mechanism in both vLLM and NVIDIA Dynamo. The decode worker picks up and generates the response.
The Numbers From Production
Baseten switched to disaggregated serving on Qwen3 Coder 480B with approximately 50,000-token prompts. They measured a 50% reduction in TTFT, a 61% increase in requests per second, and a 62% increase in tokens per second. That's not a benchmark in a controlled lab. That's a production workload at scale.
NVIDIA claims up to 7x throughput gains on Blackwell GPUs with Dynamo. I'd be skeptical of that ceiling outside of their best-case configurations, but the 2-3x gains that production teams consistently report are real and repeatable.
The cost math follows directly. If you can handle 2x the requests per second from the same GPU pool, you need half as many GPUs to serve the same load. At current H100 cloud rental pricing of roughly $2.50 to $6.50 per hour depending on provider, this adds up quickly at any meaningful scale.
What's Actually Available Today
NVIDIA Dynamo 1.0 is open source. It integrates with vLLM and SGLang as an orchestration layer, sitting above the serving runtime rather than replacing it. You don't rebuild your inference stack. You add Dynamo on top to route prefill and decode work to dedicated pools and handle the KV cache transfer between them.
If you're not ready to bring in another orchestration layer, vLLM and SGLang both support disaggregated serving natively as of their recent 2026 releases. Cursor, Perplexity, ByteDance, Meta, and LinkedIn are all running some form of disaggregated inference in production.
One thing worth knowing: the gains scale with prompt length. For short prompts (under 1,000 tokens), the overhead of the KV cache transfer between workers can eat into the benefit. Where disaggregation really shines is long-context workloads: coding agents with large codebases, RAG pipelines with big retrieved chunks, multi-turn chat with long histories. The longer your average prompt, the more you gain.
The Setup Isn't Free
This isn't a config change. Splitting into two worker pools means more infrastructure to manage: separate scaling policies for prefill and decode workers, networking that's fast enough for KV cache transfer at high request rates, and monitoring that tracks each phase independently.
I haven't run this myself at extreme scale. But the pattern makes physical sense and the production data backs it up. The thing I'd watch most carefully is KV cache transfer latency under load. If your cluster networking isn't set up for high-bandwidth inter-node traffic, the gains narrow. NIXL handles the protocol, but you still need the underlying bandwidth.
That said, at any scale where a monolithic serving setup is causing TTFT spikes or GPU utilization problems, disaggregated serving is almost certainly worth the setup cost.
Where to Start
If you're running models above 70B with long average prompt lengths, look at this seriously this quarter. Start with vLLM's native disaggregated serving mode on a staging cluster. Measure your average prefill time versus decode time per request. If prefill is taking more than 30-40% of your total latency, you're a good candidate.
The architecture is past the experimental phase. The tooling is already in your serving framework. The only thing left is deciding to turn it on.
Comments
Post a Comment