Introducing Amazon SageMaker HyperPod Inference Gateway
Amazon SageMaker HyperPod Inference Gateway is a Kubernetes‑native, GPU‑aware routing add‑on for EKS that cuts first‑token latency by up to 82% (e.g., a 4.4 s wait drops to < 800 ms) by routing requests based on real‑time GPU metrics. It uses a two‑tier design with a per‑cluster Envoy‑based gateway (Tier 1) and a forthcoming global router (Tier 2), and requires no changes to model servers or client code. Engineers can install it as a single managed add‑on and configure routing weights via an InferenceGatewayConfig CRD, gaining smarter load balancing across KV‑cache, queue depth, LoRA residency, and prefix‑cache signals.
⚡ Key Takeaways
- Latency reduction: first‑token latency improved by up to 82% (4.4 s → < 800 ms).
- Architecture: Tier 1 per‑cluster gateway built on Envoy Gateway, Body‑Based Router, and Endpoint Picker (EPP) using Prometheus metrics.
- Trade‑off: configurable scorer weights let you prioritize latency‑sensitive chat versus throughput‑optimized batch workloads.
- Integration: Deploy via the `amazon-sagemaker-hyperpod-inference` EKS managed add‑on and a declarative `InferenceGatewayConfig` custom resource.
- Limitation: Tier 2 Global Inference Router is announced as “coming soon” and not yet available.
By eliminating wasteful round‑robin routing, engineers can achieve higher GPU utilization and lower latency without over‑provisioning, directly reducing cloud spend for large‑scale LLM serving.
✅ Practical Steps
- Install the add‑on with `aws eks create-addon --cluster <cluster-name> --addon-name amazon-sagemaker-hyperpod-inference`.
- Create an `InferenceGatewayConfig` CRD specifying model routing and scorer weights to activate GPU‑aware routing.
Want the full story? Read the original article.
Read on AWS ML Blog ↗