How Salesforce is advancing global-scale model serving with cross-Region inference routing in Amazon SageMaker AI
Salesforce’s Agentforce Foundation team solved capacity fragmentation across 30+ global AWS Regions by deploying a managed, capacity‑aware cross‑Region inference router as a regional SageMaker AI resource. The router lets applications invoke a single endpoint with a routing‑target name, automatically forwarding each request to any of the listed model endpoints (identified by full ARNs) that have available capacity. This eliminates the need for custom VPC‑to‑internet traffic handling and per‑application routing logic, improving reliability while respecting data‑residency constraints. Engineers can now add capacity in any Region without redesigning their client code.
⚡ Key Takeaways
- Salesforce serves models in 30+ global Regions and hundreds of AWS accounts.
- The inference router is a regional SageMaker AI resource that routes requests based on a user‑defined routing‑target name.
- It provides capacity‑aware routing, avoiding the previous gap of no health‑signal feedback and reducing risk of cascading failures.
- Applications invoke the router with the same request body and content‑type used for InvokeEndpoint, specifying the routing target.
- The router requires the ARNs of all eligible endpoints to be listed in the routing target configuration.
🔧 Tools & Libraries
By centralizing cross‑Region routing in SageMaker, engineers can scale model capacity on demand without rewriting application logic, achieving higher availability and lower operational overhead for globally distributed AI services.
✅ Practical Steps
- Replace direct InvokeEndpoint calls with calls to the SageMaker inference router, passing the desired routing‑target name and the original request payload.
- Define a routing target by listing the full ARNs of all model endpoints that can serve the logical model (e.g., toxicity classifier) in the router’s configuration.
Want the full story? Read the original article.
Read on AWS ML Blog ↗