← Back
AWS ML Blog

SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints

•18 min read•
#deployment#inference#amazon
Level:Intermediate
For:AI Engineers
✦TL;DR

Amazon SageMaker Inference now supports GPU partitioning via NVIDIA Multi‑Instance GPU (MIG), letting a single A100 be divided into up to seven independent partitions. Each partition gets a fixed share of compute and memory with hardware‑enforced isolation, enabling multiple models per GPU and right‑sized capacity selection. The feature works only with inference components, not with single‑model, serverless, or heterogeneous‑pool endpoints. Engineers can thus increase model density and predict latency while reducing over‑provisioned GPU costs.

⚡ Key Takeaways

  • MIG can split one GPU into as many as seven instances, depending on the GPU model.
  • Partitioning is available only on endpoints that use inference components; other endpoint types cannot be partitioned.
  • Hardware isolation guarantees predictable p99 latency and fault containment between models sharing a GPU.
  • Deployments must request a GPU partition resource in the same place they request CPU and memory.
  • Single‑model endpoints, serverless endpoints, and heterogeneous instance pools are not supported for MIG partitioning.
💡 Why It Matters

By leveraging MIG through SageMaker inference components, engineers can host more models per GPU with guaranteed latency and fault isolation, cutting infrastructure spend without sacrificing performance.

✅ Practical Steps

  1. Define an inference component and specify the desired GPU partition size instead of a whole GPU.
  2. Deploy the endpoint using that inference component, which will allocate the requested MIG partition on the underlying A100.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

AWS ML Blog•#deployment

Introducing Unity Gateway Insights

Databricks Blog•#deployment

Introducing Gemini 4 Neon

Google DeepMind•#llm

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

Hugging Face Blog•#llm

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING