← Back
AWS ML Blog

Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM

•10 min read•
#inference#deployment#agents
Level:Advanced
For:AI Engineers
✦TL;DR

The post introduces bi‑directional KV cache transfer for vLLM, enabling decode nodes to send previously computed KV blocks back to prefill nodes on subsequent turns of multi‑turn agentic workloads. In 1‑prefill‑1‑decode (1P1D) benchmarks on Amazon EC2 P5en GPU instances, this technique cut time‑to‑first‑token (TTFT) by up to 1.6× for Qwen3‑32B and up to 3.3× for zai‑org/GLM‑5‑FP8. By reusing KV across turns, accelerator utilization improves and latency stays responsive as conversation histories grow.

⚡ Key Takeaways

  • TTFT reduction: up to 1.6× for Qwen3‑32B, up to 3.3× for zai‑org/GLM‑5‑FP8.
  • Architecture: bi‑directional KV transfer between prefill and decode nodes, controlled by a configurable KV recompute threshold.
  • Trade‑off: avoids redundant KV recompute on prefill nodes, reducing accelerator load and cache eviction risk.
  • Integration: uses vLLM’s accelerator‑agnostic KV transfer feature; requires AWS Elastic Fabric Adapter (EFA) with GPUDirect RDMA or Neuron Direct RDMA for direct memory moves.
  • Prerequisite: deployment on supported instances (e.g., EC2 P5en GPUs or Trainium) with EFA enabled.
💡 Why It Matters

Engineers can sustain low latency for long‑running conversational agents without over‑provisioning GPUs, directly translating to cost savings and better user experience in production AI assistants.

✅ Practical Steps

  1. Deploy a vLLM cluster with at least one prefill node and one decode node on EC2 P5en instances, ensuring each instance has AWS EFA enabled.
  2. Enable bi‑directional KV transfer in vLLM’s configuration and set the KV recompute threshold according to your workload’s turn length.
  3. Run the provided benchmark script (referenced in the post) to verify TTFT improvements for your target model.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

AWS ML Blog•#deployment

Introducing Unity Gateway Insights

Databricks Blog•#deployment

Introducing Gemini 4 Neon

Google DeepMind•#llm

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

Hugging Face Blog•#llm

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING