← Back
AWS ML Blog

Spreading the load: How Salesforce met Multi-AZ HA with SageMaker Inference Components

•11 min read•
#agents#deployment
Level:Advanced
For:AI Engineers
✦TL;DR

Salesforce achieved Multi‑AZ high availability for SageMaker Inference Components by using the new SchedulingConfig parameter in the CreateInferenceComponent API, which lets teams control copy placement across Availability Zones. By setting AvailabilityZoneBalance with MaxImbalance and using the SPREAD placement strategy, they spread four IC copies across two AZs (2 copies per AZ) while still keeping an 8× infrastructure‑cost reduction from co‑hosting models on shared GPUs. This approach satisfies Salesforce’s 2‑AZ compliance requirement without sacrificing the cost benefits of ICs.

⚡ Key Takeaways

  • 8× reduction in infrastructure costs was achieved by co‑hosting multiple models on shared GPUs.
  • SchedulingConfig with sub‑parameters AvailabilityZoneBalance and PlacementStrategy provides fine‑grained AZ and instance placement control.
  • PlacementStrategy = SPREAD distributes copies across as many instances as possible, while BINPACK packs them for higher utilization.
  • Use the CreateInferenceComponent API with SchedulingConfig to specify PlacementStrategy: SPREAD and AvailabilityZoneBalance (e.g., EnforcementMode: PERMISSIVE, MaxImbalance: 1).
  • MaxImbalance = 1 allows at most a one‑copy difference between AZs; setting MaxImbalance to 0 enforces strict equal distribution for lighter models.
💡 Why It Matters

Engineers can now meet strict Multi‑AZ HA compliance for production models while retaining the cost efficiencies of SageMaker Inference Components, enabling reliable, low‑cost AI services at scale.

✅ Practical Steps

  1. Call client.create_inference_component with a SchedulingConfig that sets PlacementStrategy to SPREAD and AvailabilityZoneBalance with MaxImbalance as needed (e.g., MaxImbalance: 1).
  2. Set RuntimeConfig CopyCount to the desired number of IC copies (e.g., 4) to have SageMaker distribute them across instances and AZs per the configured strategy.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

New in AWS Quick: More capable flows, an updated web experience, and agents that work together

AWS ML Blog•#agents

Three insights you may have missed from theCUBE’s coverage of the Dell Technologies New York Symposium

SiliconANGLE AI•#deployment

How Google Duplex evolved to also call businesses for you in Gemini

Google Research•#agents

Welcome Inkling 2 Mini, Cutting-edge Performance for your Local Setup

Hugging Face Blog•#llm

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING