Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM
The post introduces bi‑directional KV cache transfer for vLLM, enabling decode nodes to send previously computed KV blocks back to prefill nodes on subsequent turns of multi‑turn agentic workloads. In 1‑prefill‑1‑decode (1P1D) benchmarks on Amazon EC2 P5en GPU instances, this technique cut time‑to‑first‑token (TTFT) by up to 1.6× for Qwen3‑32B and up to 3.3× for zai‑org/GLM‑5‑FP8. By reusing KV across turns, accelerator utilization improves and latency stays responsive as conversation histories grow.
⚡ Key Takeaways
- TTFT reduction: up to 1.6× for Qwen3‑32B, up to 3.3× for zai‑org/GLM‑5‑FP8.
- Architecture: bi‑directional KV transfer between prefill and decode nodes, controlled by a configurable KV recompute threshold.
- Trade‑off: avoids redundant KV recompute on prefill nodes, reducing accelerator load and cache eviction risk.
- Integration: uses vLLM’s accelerator‑agnostic KV transfer feature; requires AWS Elastic Fabric Adapter (EFA) with GPUDirect RDMA or Neuron Direct RDMA for direct memory moves.
- Prerequisite: deployment on supported instances (e.g., EC2 P5en GPUs or Trainium) with EFA enabled.
Engineers can sustain low latency for long‑running conversational agents without over‑provisioning GPUs, directly translating to cost savings and better user experience in production AI assistants.
✅ Practical Steps
- Deploy a vLLM cluster with at least one prefill node and one decode node on EC2 P5en instances, ensuring each instance has AWS EFA enabled.
- Enable bi‑directional KV transfer in vLLM’s configuration and set the KV recompute threshold according to your workload’s turn length.
- Run the provided benchmark script (referenced in the post) to verify TTFT improvements for your target model.
Want the full story? Read the original article.
Read on AWS ML Blog ↗