Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod
Amazon SageMaker HyperPod now adds three resiliency capabilities—MultiNodeTrainingValidation, a Kubernetes Resource Monitoring Agent, and Hung job detection—to catch silent GPU faults, Kubernetes component failures, and stalled Slurm jobs earlier, thereby improving cluster goodput. MultiNodeTrainingValidation trains a small model across provisioned instances and flags NaN, infinite loss, or timeout failures. The Kubernetes Resource Monitoring Agent surfaces device‑plugin crashes, missing DaemonSets, and tainted nodes as CloudWatch events, while Hung job detection monitors GPU activity and job output to auto‑cancel and requeue stalled jobs. These tools give engineers concrete signals to intervene before compute is wasted.
⚡ Key Takeaways
- MultiNodeTrainingValidation runs a small training job on each provisioned instance and fails if loss becomes NaN, infinite, or the job exceeds its time budget.
- The Kubernetes Resource Monitoring Agent emits Kubernetes events and CloudWatch logs for crash‑looping device plugins, missing DaemonSets, and tainted nodes.
- Hung job detection watches per‑node job output, GPU utilization, and user‑defined log patterns, writing CloudWatch events and optionally canceling the Slurm job for requeue.
- All three features are configurable on demand or during instance provisioning in Amazon EKS‑managed HyperPod clusters.
- They complement existing health checks (HMA, InstanceStress, InstanceConnectivity) which only verify hardware at the node level.
By surfacing silent data corruption, Kubernetes misconfigurations, and stalled training jobs in real time, engineers can reclaim wasted GPU hours and maintain higher training throughput in production clusters.
✅ Practical Steps
- Enable MultiNodeTrainingValidation on a HyperPod cluster via the provided deep health check command during provisioning or on demand.
- Deploy the HyperPod Kubernetes Resource Monitoring Agent to emit events for device‑plugin crashes, missing DaemonSets, and tainted nodes to CloudWatch.
- Activate Hung job detection by configuring the per‑node agent with desired log patterns; it will write CloudWatch events and can auto‑cancel stalled Slurm jobs.
Want the full story? Read the original article.
Read on AWS ML Blog ↗