Prefill Decode Disaggregation: Latency, Cost & 2026 Trade-offs

A 64k-token document lands in a busy chat service. Average throughput still looks healthy, but time to first token spikes because the GPU is processing that large prompt while other users are waiting for their next decode step.

That is the problem prefill–decode disaggregation tries to solve. It assigns prompt processing and token generation to separate worker pools so a burst of long inputs doesn’t interrupt active conversations.

The architecture can reduce tail latency. It can also double some GPU capacity, add network traffic, and create another class of failure between request routing and generation. The practical choice is therefore not “disaggregated serving is faster.” It is whether the latency improvement justifies the extra infrastructure for a particular workload.

Start with a unified serving pool. Split prefill and decode only after production measurements show that phase interference is hurting your SLO.

Why Separate Prefill From Decode?

An LLM request has two distinct phases.

Prefill processes the input prompt. It is highly parallel and often compute-intensive. A 40,000-token document can consume substantial GPU capacity in a short burst.

Decode generates the response one token at a time. It is sequential, memory-bandwidth-sensitive, and directly visible to the user through inter-token latency.

On a unified server, both phases compete for scheduling slots and GPU memory. A long prompt can delay the first token for requests that are already generating. Average latency may remain acceptable while p95 and p99 time to first token deteriorate.

With disaggregation:

1. A router sends new prompts to prefill workers.
2. Prefill workers build the KV cache.
3. The relevant cache state moves to decode workers.
4. Decode workers generate responses without sharing their queue with new long prompts.

That pattern fits retrieval-augmented generation, document analysis, coding agents, and chat systems with a wide range of input lengths. It is less attractive for short prompts, low concurrency, or workloads dominated by long outputs.

The handoff is the central trade-off. KV state must cross a network boundary, and both pools need enough spare capacity to absorb bursts. High-speed interconnects and careful scheduling can make that worthwhile. Ordinary Ethernet can simply move the bottleneck from the GPU scheduler to the network.

A Worked Latency Example

Consider a request whose 32k-token prompt takes 180 ms to process on a unified worker. It then waits 120 ms behind other prefills before decode begins. The total TTFT is roughly:

• Prompt processing: 180 ms
• Queueing behind prefill work: 120 ms
• KV transfer: 0 ms because the phases share a worker
• Total TTFT: 300 ms

After disaggregation, the prefill pool processes the prompt in 180 ms. The KV cache takes 25 ms to reach a decode worker, and the decode queue contributes another 35 ms:

• Prompt processing: 180 ms
• KV transfer: 25 ms
• Decode queueing: 35 ms
• Total TTFT: 240 ms

The split improves TTFT by 20 percent, but it has not made prefill faster. It has removed 85 ms of contention. If the transfer takes 180 ms on a slower network, the same design becomes a regression.

vLLM, SGLang, or TensorRT-LLM?

The major runtimes now overlap on core capabilities: low-precision KV caches, speculative decoding, mixture-of-experts execution, multimodal models, and disaggregated serving. Their differences are more visible in hardware coverage, release velocity, and how much tuning the platform team is prepared to own.

The version-specific claims below refer to project release material dated October 18–19, 2026: vLLM 0.32.0 release notes and SGLang 0.5.22 release notes. TensorRT-LLM capabilities should be checked against the current NVIDIA documentation, since its supported architectures and engine features change quickly.

The SGLang figures are useful signals, not neutral head-to-head results. A fair comparison must hold the model, quantization, GPU, tensor parallelism, prompt distribution, concurrency, KV precision, and latency target constant. Otherwise, “27% faster” may describe two different experiments rather than two runtimes.

vLLM is usually the least disruptive starting point when a platform supports several models or GPU generations. SGLang merits a serious benchmark for long-context, MoE-heavy, or scheduling-sensitive traffic. TensorRT-LLM is a strong candidate for a stable NVIDIA fleet where engine optimization work can be amortized across high request volume.

KV Cache, Memory, and Cost

KV-cache memory is a major constraint in long-context serving. A simplified estimate is:

KV memory: 2 × layers × cached tokens × KV heads × head dimension × bytes per element

That formula needs an important qualification. In grouped-query or multi-query attention, use the number of key/value heads, not the number of query heads. Tensor parallelism, page size, allocator overhead, batching, and runtime metadata add further memory use. Treat the equation as a capacity estimate, not an exact allocation figure.

Moving from FP16 to FP8 roughly halves the raw KV-cache footprint. That can support longer contexts, more active sequences, and fewer evictions. It does not halve the cost of serving: model weights, activations, network capacity, and idle headroom remain.

For a simple cost example, suppose a four-GPU replica costs $12 per GPU-hour, or $48 per hour. At 1 million output tokens per hour, the GPU cost is $48 per million output tokens. A genuine 27 percent throughput increase, with the same hardware and utilization, would reduce that figure to about $37.80.

Now add a second four-GPU pool for disaggregation. The nominal GPU bill doubles. The architecture is financially attractive only if it prevents enough queueing, raises useful utilization, or lets the operator use cheaper, better-matched hardware for one phase.

Measure the split against a fixed request-success rate and p95 target. At minimum, capture:

• p50 and p95 TTFT;
• decode time per output token;
• prompt and completion throughput;
• GPU utilization by phase;
• KV-cache hit and eviction rates;
• KV-transfer latency and network bytes.

Rejected, timed-out, and cancelled requests must remain in the accounting. A system that produces impressive tokens per second while failing more requests is not a production win.

Use realistic prompt buckets in the benchmark. A traffic mix containing 50 percent 1k–4k-token prompts, 35 percent 8k–16k prompts, and 15 percent 32k–64k prompts will expose behavior that a uniform 4k test hides.

A Practical Go/No-Go Test

First, establish a unified baseline with the production model, quantization, hardware, concurrency, client behavior, and prompt distribution. Then run the disaggregated design under the same conditions.

A baseline server might look like this:

vllm serve /models/your-model \
  --tensor-parallel-size 4 \
  --kv-cache-dtype fp8 \
  --max-model-len 32768 \
  --enable-prefix-caching

The flags are illustrative. Validate output quality, structured responses, tool calls, streaming cancellation, and multimodal behavior after enabling low-precision or experimental features.

A useful go/no-go rule is:

1. Hold the request-success rate constant, such as 99.5 percent.
2. Require at least a 20 percent improvement in measured p95 TTFT against the unified baseline, or an equivalent reduction in GPUs needed to meet the same SLO.
3. Calculate the added monthly GPU, network, storage, and operational cost.
4. Proceed only if the value of the latency improvement or capacity reduction exceeds that added cost with a reasonable margin.

For example, if the split lowers p95 TTFT by 24 percent at the same 99.5 percent success rate but doubles GPU cost, it probably fails the business case unless the product has a high latency premium. If it achieves the same p95 target with fewer decode GPUs, the calculation may favor disaggregation even when raw throughput barely changes.

Stay unified when prompts are generally below 4k–8k tokens, concurrency is modest, or the split leaves one pool mostly idle. A well-packed shared pool can outperform a more sophisticated design that spends most of its time waiting for work.

Speculative decoding deserves the same measurement discipline. Track draft-token acceptance, TTFT, decode time, output quality, and total utilization together. It can help when acceptance is high; it can hurt when requests are short, outputs are highly stochastic, or verification pressure damages batching.

API compatibility also goes beyond a successful chat completion. Test tool calls, JSON or grammar-constrained output, multimodal inputs, log probabilities, prefix caching, retries, timeouts, and partial-stream failures. vLLM and SGLang may expose similar APIs while differing in edge-case behavior.

The right runtime is the one that meets the latency target at acceptable cost under the actual prompt distribution. Benchmark that workload first, then decide whether the architecture needs a split.