← Back
AWS ML Blog

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

•30 min read•
#deployment#amazon
Level:Advanced
For:AI Engineers
✦TL;DR

Amazon SageMaker HyperPod now adds three resiliency capabilities—MultiNodeTrainingValidation, a Kubernetes Resource Monitoring Agent, and Hung job detection—to catch silent GPU faults, Kubernetes component failures, and stalled Slurm jobs earlier, thereby improving cluster goodput. MultiNodeTrainingValidation trains a small model across provisioned instances and flags NaN, infinite loss, or timeout failures. The Kubernetes Resource Monitoring Agent surfaces device‑plugin crashes, missing DaemonSets, and tainted nodes as CloudWatch events, while Hung job detection monitors GPU activity and job output to auto‑cancel and requeue stalled jobs. These tools give engineers concrete signals to intervene before compute is wasted.

⚡ Key Takeaways

  • MultiNodeTrainingValidation runs a small training job on each provisioned instance and fails if loss becomes NaN, infinite, or the job exceeds its time budget.
  • The Kubernetes Resource Monitoring Agent emits Kubernetes events and CloudWatch logs for crash‑looping device plugins, missing DaemonSets, and tainted nodes.
  • Hung job detection watches per‑node job output, GPU utilization, and user‑defined log patterns, writing CloudWatch events and optionally canceling the Slurm job for requeue.
  • All three features are configurable on demand or during instance provisioning in Amazon EKS‑managed HyperPod clusters.
  • They complement existing health checks (HMA, InstanceStress, InstanceConnectivity) which only verify hardware at the node level.
💡 Why It Matters

By surfacing silent data corruption, Kubernetes misconfigurations, and stalled training jobs in real time, engineers can reclaim wasted GPU hours and maintain higher training throughput in production clusters.

✅ Practical Steps

  1. Enable MultiNodeTrainingValidation on a HyperPod cluster via the provided deep health check command during provisioning or on demand.
  2. Deploy the HyperPod Kubernetes Resource Monitoring Agent to emit events for device‑plugin crashes, missing DaemonSets, and tainted nodes to CloudWatch.
  3. Activate Hung job detection by configuring the per‑node agent with desired log patterns; it will write CloudWatch events and can auto‑cancel stalled Slurm jobs.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

Introducing Unity Gateway Insights

Databricks Blog•#deployment

Introducing Gemini 4 Neon

Google DeepMind•#llm

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

Hugging Face Blog•#llm

Mathematical proof, powered by AI. And no hallucinations!

Amazon Science•#agents

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING