Introducing enhanced infrastructure observability in Amazon SageMaker HyperPod
Amazon SageMaker HyperPod now includes enhanced observability that automatically collects default metrics every 30 seconds with ~1–2 % CPU overhead and 250–300 MB memory, and offers opt‑in advanced metrics such as full NVIDIA DCGM, AWS Neuron, and EFA telemetry. The system uses CloudWatch’s DetectGroupOutliers API to flag GPUs deviating on utilization, memory, temperature, power, and SM clock, and applies static rules to surface XID errors, ECC faults, and thermal throttling. Engineers can route this telemetry to either Amazon CloudWatch (new default, lower‑cost) or Amazon Managed Service for Prometheus, enabling faster root‑cause diagnosis and reduced GPU waste.
⚡ Key Takeaways
- Default metrics are emitted every 30 seconds with ~1–2 % CPU overhead and 250–300 MB memory usage.
- Advanced metrics (opt‑in) include the full NVIDIA GPU DCGM suite, AWS Neuron metrics, and Elastic Fabric Adapter (EFA) statistics.
- CloudWatch DetectGroupOutliers API automatically flags outlier GPUs based on utilization, memory, temperature, power, and SM clock.
- Metric stores can be Amazon CloudWatch (default) or Amazon Managed Service for Prometheus + Grafana.
- Advanced metrics increase telemetry volume and therefore incur higher storage costs.
By surfacing hardware‑level failures and network bottlenecks in near‑real time, engineers can cut troubleshooting time from days to minutes, keeping expensive accelerators productive and lowering overall training cost.
✅ Practical Steps
- Enable SageMaker HyperPod default observability (no action required; it runs automatically).
- Opt‑in to advanced metrics via the HyperPod configuration to collect DCGM, Neuron, and EFA data.
- Choose your metric store: set the destination to Amazon CloudWatch for native SageMaker Insights or to Amazon Managed Service for Prometheus for Grafana dashboards.
Want the full story? Read the original article.
Read on AWS ML Blog ↗