Spreading the load: How Salesforce met Multi-AZ HA with SageMaker Inference Components
Salesforce achieved Multi‑AZ high availability for SageMaker Inference Components by using the new SchedulingConfig parameter in the CreateInferenceComponent API, which lets teams control copy placement across Availability Zones. By setting AvailabilityZoneBalance with MaxImbalance and using the SPREAD placement strategy, they spread four IC copies across two AZs (2 copies per AZ) while still keeping an 8× infrastructure‑cost reduction from co‑hosting models on shared GPUs. This approach satisfies Salesforce’s 2‑AZ compliance requirement without sacrificing the cost benefits of ICs.
⚡ Key Takeaways
- 8× reduction in infrastructure costs was achieved by co‑hosting multiple models on shared GPUs.
- SchedulingConfig with sub‑parameters AvailabilityZoneBalance and PlacementStrategy provides fine‑grained AZ and instance placement control.
- PlacementStrategy = SPREAD distributes copies across as many instances as possible, while BINPACK packs them for higher utilization.
- Use the CreateInferenceComponent API with SchedulingConfig to specify PlacementStrategy: SPREAD and AvailabilityZoneBalance (e.g., EnforcementMode: PERMISSIVE, MaxImbalance: 1).
- MaxImbalance = 1 allows at most a one‑copy difference between AZs; setting MaxImbalance to 0 enforces strict equal distribution for lighter models.
Engineers can now meet strict Multi‑AZ HA compliance for production models while retaining the cost efficiencies of SageMaker Inference Components, enabling reliable, low‑cost AI services at scale.
✅ Practical Steps
- Call client.create_inference_component with a SchedulingConfig that sets PlacementStrategy to SPREAD and AvailabilityZoneBalance with MaxImbalance as needed (e.g., MaxImbalance: 1).
- Set RuntimeConfig CopyCount to the desired number of IC copies (e.g., 4) to have SageMaker distribute them across instances and AZs per the configured strategy.
Want the full story? Read the original article.
Read on AWS ML Blog ↗