← Back
AWS ML Blog

Amazon SageMaker AI real-time endpoints now support KV cache aware routing

•22 min read•
#deployment#llm
Level:Advanced
For:AI Engineers
✦TL;DR

Amazon SageMaker AI adds KV‑cache‑aware routing (KV_AWARE) for real‑time endpoints, allowing the router to direct each request to an instance that already holds the matching KV cache and isn’t overloaded. The model container reports cache events via the SM_VLLM_KV_EVENTS_CONFIG environment variable, and the endpoint’s routing strategy is set to KV_AWARE in the endpoint configuration. This improves latency for multi‑turn LLM chats, RAG, and agents with long shared prompts by avoiding unnecessary recomputation of cached prefixes. Engineers can enable it with a single env‑var and routing flag while keeping the same InvokeEndpoint API.

⚡ Key Takeaways

  • KV‑aware routing is enabled by setting the environment variable SM_VLLM_KV_EVENTS_CONFIG in the model container and configuring RoutingStrategy=KV_AWARE on the endpoint.
  • The router uses a managed cache index of one‑way truncated hash pointers that map cached token prefixes to specific instances or inference component copies.
  • It avoids the stale‑cache problem of prefix‑aware routing by considering real‑time cache evictions reported by the container.
  • Requests are tokenized, matched against the index, and routed to the instance holding the longest cached prefix while respecting instance load.
  • The feature works for both single‑model endpoints and inference component endpoints; verification is done by inspecting cache events and routing decisions.
💡 Why It Matters

By routing to warm‑cache instances, engineers can cut inference latency and GPU compute waste for conversational LLM services, directly translating to lower cost and higher throughput in production.

✅ Practical Steps

  1. Add SM_VLLM_KV_EVENTS_CONFIG to the model container’s environment so it publishes KV cache store/evict events.
  2. In the SageMaker endpoint configuration, set RoutingStrategy=KV_AWARE to activate KV‑aware routing.
  3. Deploy the endpoint and invoke it with the standard InvokeEndpoint or OpenAI‑compatible Chat Completions API; the router will now use the cache index to select the optimal instance.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

AWS ML Blog•#deployment

Introducing Unity Gateway Insights

Databricks Blog•#deployment

Introducing Gemini 4 Neon

Google DeepMind•#llm

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

Hugging Face Blog•#llm

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING