← Back
AWS ML Blog

Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM

•10 min read•
#inference#llm#deployment
Level:Advanced
For:AI Engineers
✦TL;DR

vLLM’s new push‑based NIXL connector moves KV‑cache transfer off the critical path of disaggregated prefill/decode inference, cutting time‑to‑first‑token (TTFT) by up to 3× and improving time‑per‑output‑token (TPOT) by up to 30 % for the Qwen3‑32B model on an Amazon EC2 P5en GPU cluster with AWS Elastic Fabric Adapter (EFA). The design overlaps producer (prefill) and consumer (decode) work by pre‑registering target blocks and using RDMA‑direct KV moves, eliminating the sequential router‑relay, allocation, and NIXL READ steps of the prior pull mode. Engineers can enable push mode in vLLM to achieve lower latency and higher throughput under high request rates without changing model code.

⚡ Key Takeaways

  • Push‑mode KV transfer achieved up to 3× better TTFT and up to 30 % better TPOT versus pull mode for Qwen3‑32B.
  • The architecture registers decode‑side target blocks in advance and uses the NVIDIA Inference Xfer Library (NIXL) over RDMA (AWS EFA) to push KV blocks directly between GPU (or Trainium) memories.
  • Overlapping producer and consumer work removes the serialization bottleneck, reducing latency at high request rates.
  • Enable the mode via vLLM’s NIXL connector configuration (push mode flag) to use the accelerator‑agnostic push‑based transfer.
  • Requires instances with RDMA‑capable interconnects (e.g., AWS EC2 P5en with EFA or Trainium instances supporting GPUDirect/Neuron Direct RDMA).
💡 Why It Matters

Reducing TTFT and TPOT directly improves user‑perceived responsiveness and overall throughput for large‑scale LLM serving, enabling cost‑effective scaling of disaggregated inference pipelines.

✅ Practical Steps

  1. Set the NIXL connector to push mode in the vLLM configuration when launching a disaggregated prefill/decode deployment.
  2. Deploy the inference service on RDMA‑enabled instances (e.g., EC2 P5en with AWS EFA) to allow direct GPU‑to‑GPU KV transfers.
  3. Run the provided benchmark script to verify TTFT and TPOT improvements on your workload.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

AWS ML Blog•#deployment

Introducing Unity Gateway Insights

Databricks Blog•#deployment

Introducing Gemini 4 Neon

Google DeepMind•#llm

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

Hugging Face Blog•#llm

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING