Multi-Region training with Amazon SageMaker HyperPod and Qumulo
Amazon SageMaker HyperPod paired with Qumulo’s Cloud Native storage enables cross‑Region training without data replication, achieving 115–117 samples/sec on a remote US‑West (Oregon) cluster despite 60 ms network latency. After a brief warm‑up (under 1 % of wall‑clock time for 10,000+ batches), throughput matches a co‑located hub cluster, delivering 98–100 % GPU utilization. The solution leverages Qumulo’s Cloud Data Fabric with NeuralCache to cache data locally on NVMe, making cross‑Region latency transparent. Engineers can now keep frontier models current without moving petabytes of data or sacrificing performance.
⚡ Key Takeaways
- Remote HyperPod cluster achieved 115–117 samples/sec, matching hub performance.
- Uses Qumulo’s Cloud Data Fabric (CDF) with NeuralCache for predictive NVMe caching.
- Warm‑up overhead is under 1 % of wall‑clock time for 10,000+ batches (under 0.1 % for >100,000 batches).
- Each cluster mounts its local CNQ instance via NFS and accesses the hub’s data over VPC peering.
- Full 99 % GPU utilization and p5.48xlarge network saturation with sub‑3 ms data operations on the hub side.
This architecture eliminates costly data duplication and latency penalties, letting production teams scale training across regions while maintaining near‑optimal GPU utilization and throughput.
✅ Practical Steps
- Store the training dataset in a Cloud Native Qumulo (CNQ) instance in a single AWS Region.
- In each SageMaker HyperPod cluster, mount the local CNQ instance over NFS and configure VPC peering to the hub CNQ for remote data access.
- Rely on Qumulo’s Cloud Data Fabric (CDF) with NeuralCache to automatically cache accessed data on local NVMe, ensuring cross‑Region latency becomes negligible after the initial warm‑up period.
Want the full story? Read the original article.
Read on AWS ML Blog ↗