Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine

The article introduces a tiered key‑value (KV) cache for large language models running on Amazon SageMaker HyperPod, leveraging Curvine’s distributed NVMe pool to extend cache capacity beyond GPU memory. By layering GPU‑resident cache with a shared NVMe tier, the approach cuts first‑token latency while keeping memory footprints manageable, enabling higher throughput without scaling to oversized GPU instances. The design trades off modest storage costs for significant speed gains, making it attractive for production deployments that demand low‑latency inference at scale.

⚡ Key Takeaways

• The tiered cache architecture couples on‑device GPU memory with a Curvine‑managed NVMe pool, allowing replicas to share cached activations across the HyperPod cluster.
• Curvine’s distributed NVMe pool is configured as a secondary cache tier, reducing the need for larger GPU instances while keeping data locality high.
• The tradeoff is an increased storage footprint and associated cost, but the system delivers lower time‑to‑first‑token and higher overall throughput for large LLM inference workloads.
• Integration requires enabling the tiered KV cache in the SageMaker HyperPod configuration and pointing the cache backend to the Curvine NVMe pool endpoint.
• The solution assumes the presence of Cur

Want the full story? Read the original article.