SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints
Amazon SageMaker Inference now supports GPU partitioning via NVIDIA Multi‑Instance GPU (MIG), letting a single A100 be divided into up to seven independent partitions. Each partition gets a fixed share of compute and memory with hardware‑enforced isolation, enabling multiple models per GPU and right‑sized capacity selection. The feature works only with inference components, not with single‑model, serverless, or heterogeneous‑pool endpoints. Engineers can thus increase model density and predict latency while reducing over‑provisioned GPU costs.
⚡ Key Takeaways
- MIG can split one GPU into as many as seven instances, depending on the GPU model.
- Partitioning is available only on endpoints that use inference components; other endpoint types cannot be partitioned.
- Hardware isolation guarantees predictable p99 latency and fault containment between models sharing a GPU.
- Deployments must request a GPU partition resource in the same place they request CPU and memory.
- Single‑model endpoints, serverless endpoints, and heterogeneous instance pools are not supported for MIG partitioning.
By leveraging MIG through SageMaker inference components, engineers can host more models per GPU with guaranteed latency and fault isolation, cutting infrastructure spend without sacrificing performance.
✅ Practical Steps
- Define an inference component and specify the desired GPU partition size instead of a whole GPU.
- Deploy the endpoint using that inference component, which will allocate the requested MIG partition on the underlying A100.
Want the full story? Read the original article.
Read on AWS ML Blog ↗