← Back
AWS ML Blog

Introducing Amazon SageMaker HyperPod Global Inference Gateway: One endpoint across Regions and clusters

•13 min read•
#deployment#amazon#inference
Level:Advanced
For:AI Engineers
✦TL;DR

Amazon SageMaker HyperPod now offers a Global Inference Gateway that provides a single, SageMaker‑managed, GPU‑aware endpoint to route requests across all registered HyperPod clusters, even across AWS Regions. It automatically selects the best‑suited cluster using real‑time signals (KV‑cache utilization, queue depth), handles cross‑Region failover, and authenticates via SigV4 and IAM over PrivateLink. This reduces endpoint management from 20 to 5 in a typical four‑model, five‑cluster, two‑Region setup and improves GPU utilization and reliability for production inference workloads.

⚡ Key Takeaways

  • Reduces endpoint count from 20 to 5 by consolidating to one endpoint per cluster.
  • Uses real‑time KV‑cache utilization and queue‑depth signals to pick the optimal pod inside each cluster.
  • Provides automatic cross‑Region failover, moving traffic away from a failed cluster without client changes.
  • Integrates with AWS Signature Version 4 and IAM for authentication, and traffic travels over AWS PrivateLink.
  • Requires registration of each HyperPod cluster with the Global Inference Gateway; unregistered clusters will not receive traffic.
💡 Why It Matters

Engineers can maximize existing GPU capacity, eliminate custom load‑balancing and DNS plumbing, and achieve seamless cross‑Region resilience with a single endpoint, accelerating time‑to‑value for large‑scale LLM inference services.

✅ Practical Steps

  1. Register each SageMaker HyperPod cluster with the Global Inference Gateway as described in the SageMaker documentation.
  2. Update client applications to call the single SageMaker‑managed endpoint instead of multiple per‑cluster or per‑model endpoints.
  3. Verify that IAM policies and SigV4 signing are configured so requests authenticate and travel over AWS PrivateLink.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

New in AWS Quick: More capable flows, an updated web experience, and agents that work together

AWS ML Blog•#agents

Three insights you may have missed from theCUBE’s coverage of the Dell Technologies New York Symposium

SiliconANGLE AI•#deployment

Welcome Inkling 2 Mini, Cutting-edge Performance for your Local Setup

Hugging Face Blog•#llm

NVIDIA and Microsoft Expand Accelerated Computing From Azure to Windows PCs

NVIDIA Blog•#compute

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING