← Back
AWS ML Blog

Introducing enhanced infrastructure observability in Amazon SageMaker HyperPod

•12 min read•
#deployment
Level:Advanced
For:AI Engineers
✦TL;DR

Amazon SageMaker HyperPod now includes enhanced observability that automatically collects default metrics every 30 seconds with ~1–2 % CPU overhead and 250–300 MB memory, and offers opt‑in advanced metrics such as full NVIDIA DCGM, AWS Neuron, and EFA telemetry. The system uses CloudWatch’s DetectGroupOutliers API to flag GPUs deviating on utilization, memory, temperature, power, and SM clock, and applies static rules to surface XID errors, ECC faults, and thermal throttling. Engineers can route this telemetry to either Amazon CloudWatch (new default, lower‑cost) or Amazon Managed Service for Prometheus, enabling faster root‑cause diagnosis and reduced GPU waste.

⚡ Key Takeaways

  • Default metrics are emitted every 30 seconds with ~1–2 % CPU overhead and 250–300 MB memory usage.
  • Advanced metrics (opt‑in) include the full NVIDIA GPU DCGM suite, AWS Neuron metrics, and Elastic Fabric Adapter (EFA) statistics.
  • CloudWatch DetectGroupOutliers API automatically flags outlier GPUs based on utilization, memory, temperature, power, and SM clock.
  • Metric stores can be Amazon CloudWatch (default) or Amazon Managed Service for Prometheus + Grafana.
  • Advanced metrics increase telemetry volume and therefore incur higher storage costs.
💡 Why It Matters

By surfacing hardware‑level failures and network bottlenecks in near‑real time, engineers can cut troubleshooting time from days to minutes, keeping expensive accelerators productive and lowering overall training cost.

✅ Practical Steps

  1. Enable SageMaker HyperPod default observability (no action required; it runs automatically).
  2. Opt‑in to advanced metrics via the HyperPod configuration to collect DCGM, Neuron, and EFA data.
  3. Choose your metric store: set the destination to Amazon CloudWatch for native SageMaker Insights or to Amazon Managed Service for Prometheus for Grafana dashboards.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

New in AWS Quick: More capable flows, an updated web experience, and agents that work together

AWS ML Blog•#agents

Three insights you may have missed from theCUBE’s coverage of the Dell Technologies New York Symposium

SiliconANGLE AI•#deployment

Welcome Inkling 2 Mini, Cutting-edge Performance for your Local Setup

Hugging Face Blog•#llm

NVIDIA and Microsoft Expand Accelerated Computing From Azure to Windows PCs

NVIDIA Blog•#compute

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING