---
title: "vLLM's 3x latency improvement comes with an earlier routing commitment"
description: "AWS demonstrates up to threefold lower first-token latency using push-based KV transfer in vLLM. Platform teams must benchmark routing flexibility, failure handling and full serving cost before switching."
url: "https://devencelab.com/insights/2026/10/19/vllms-3x-latency-improvement-comes-with-an-earlier-routing"
date: "2026-10-19"
section: "Insights"
tag: "GPU & Compute"
author: "Devence Lab"
reading_time: "3 min read"
site: "Devence Lab"
license: "Readable and quotable with attribution to the canonical URL."
---

# vLLM's 3x latency improvement comes with an earlier routing commitment

AWS demonstrates up to threefold lower first-token latency using push-based KV transfer in vLLM. Platform teams must benchmark routing flexibility, failure handling and full serving cost before switching.

AWS reported on 19 October 2026 that a push-based NIXL connector for vLLM reduced time-to-first-token by up to roughly threefold and time-per-output-token by as much as 30% in a benchmark of disaggregated inference. The improvement is not a new model capability. It comes from moving key-value cache transfer setup out of the decoder's latency-critical path. The design also commits to a decode destination earlier, creating an orchestration trade-off that production teams must measure.

In prefill-decode disaggregation, one worker processes the prompt and constructs the key-value cache while another generates tokens. The cache must reach the decode worker before generation can start. AWS explains that a traditional pull-mode connector makes the decode worker wait for metadata, allocate blocks, prepare the transfer and request the data after prefill completes.

## Push removes serial work, not the cache transfer

In push mode the decoder registers destination blocks while the prefill worker is still computing. The producer matches that registration with completed cache data and transfers it using an NIXL WRITE from a background thread. Notifications indicate completion before decoding begins. This overlaps setup with computation and avoids blocking the decode engine loop during descriptor preparation.

The design handles registrations that arrive before or after prefill and matches requests even across retries or pre-emption. Operators still need to test what happens when a worker fails after the decode node reserves memory, or when a transfer times out under contention. Lower latency is not the same as reliable recovery.

> A faster first token is useful only when early worker selection does not undermine queue fairness, capacity management or recovery.

## The threefold result describes a specific bottleneck

AWS used Qwen3-32B on one prefill and one decode instance, Amazon EC2 P5en nodes connected by Elastic Fabric Adapter, 128 output tokens and prompt lengths from 1,000 to 16,000 tokens. Prefix caching was deliberately disabled to expose full cache-transfer costs. At eight queries per second with 16,000-token prompts, pull-mode first-token latency reached about 1,744 milliseconds versus roughly 540 milliseconds for push mode.

At low traffic and short prompts, the reported difference was small. Results depend on instance type, model, network, cache reuse and scheduler behaviour. Benchmarking only peak throughput would miss the workload region where the new connector actually helps.

## Routing policy decides whether push is appropriate

Pull mode lets the router choose a decode node after prefill, preserving flexibility for load-aware placement and quality-of-service decisions. Push mode needs that destination selected earlier to overlap registration with computation. For heterogeneous clusters or volatile demand, the placement cost may outweigh the transfer gain.

Measure p50 and p95 first-token latency, output-token latency, request cancellations, queue time, cache transfer failures and cost per completed request. Run both modes against the same traffic mix and hardware, including prefix-cache workloads. AWS advises keeping pull for cases that need late decoder selection or bidirectional transfer. This is a choice about the whole serving architecture, not a universal threefold acceleration switch.

Produced through Devence Lab's automated research and publishing workflow, without prepublication human review. Devence Lab retains editorial responsibility. Sources are linked; report factual errors to contact@devencelab.com.

## Sources

- [Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM](https://aws.amazon.com/blogs/machine-learning/push-based-kv-transfer-to-improve-ttft-and-tpot-for-disaggregated-inference-in-vllm/) - AWS Machine Learning Blog