Fault tolerant distributed training on Amazon EKS using NVRx

The article demonstrates how NVIDIA Resiliency Extension (NVRx) can be integrated into PyTorch Fully Sharded Data Parallel (FSDP) training on Amazon EKS, enabling asynchronous checkpoint I/O that overlaps with training and allowing in‑process GPU fault recovery within seconds. Benchmarks on H100 GPUs across 2 to 8 nodes show that this approach maintains near‑linear scaling while providing robust fault tolerance. The solution leverages ft_launcher’s in‑job restart capability and async checkpointing hooks to minimize downtime. The trade‑off is a modest increase in checkpoint overhead, but the overall training throughput remains high.

⚡ Key Takeaways

• NVRx enables asynchronous checkpointing that overlaps with training, reducing idle time during fault recovery.
• ft_launcher’s in‑job restart mode allows the training job to resume automatically after a GPU failure without restarting the entire cluster.
• The approach was benchmarked on H100 GPUs from 2 to 8 nodes, demonstrating near‑linear scaling while handling GPU faults in seconds.
• Integration requires adding the NVRx library to the training container and configuring ft_launcher with the `--restart-mode=on-failure` flag.
• The method assumes EKS nodes are provisioned with H100 GPUs and that the training script uses PyTorch FSDP with the `checkpointing=True` option.
• WhyItMatters: Engineers deploying large‑scale distributed training on Kubernetes can now achieve high availability without sacrificing performance, which is critical for production workloads that demand minimal downtime.
• TechnicalLevel: Advanced
• TargetAudience: ML Engineers, Distributed Training Engineers
• PracticalSteps:
• Add the NVIDIA Resiliency Extension to your Docker image and import it in your training script.
• Run the training job with ft_launcher using `--restart-mode=on-failure` to enable in

Engineers deploying large‑scale distributed training on Kubernetes can now achieve high availability without sacrificing performance, which is critical for production workloads that demand minimal downtime.

✅ Practical Steps

1. Add the NVIDIA Resiliency Extension to your Docker image and import it in your training script.
2. Run the training job with ft_launcher using `--restart-mode=on-failure` to enable in

Want the full story? Read the original article.