← Back
AWS ML Blog

Multi-Region training with Amazon SageMaker HyperPod and Qumulo

•18 min read•
#deployment#compute
Level:Advanced
For:AI Engineers
✦TL;DR

Amazon SageMaker HyperPod paired with Qumulo’s Cloud Native storage enables cross‑Region training without data replication, achieving 115–117 samples/sec on a remote US‑West (Oregon) cluster despite 60 ms network latency. After a brief warm‑up (under 1 % of wall‑clock time for 10,000+ batches), throughput matches a co‑located hub cluster, delivering 98–100 % GPU utilization. The solution leverages Qumulo’s Cloud Data Fabric with NeuralCache to cache data locally on NVMe, making cross‑Region latency transparent. Engineers can now keep frontier models current without moving petabytes of data or sacrificing performance.

⚡ Key Takeaways

  • Remote HyperPod cluster achieved 115–117 samples/sec, matching hub performance.
  • Uses Qumulo’s Cloud Data Fabric (CDF) with NeuralCache for predictive NVMe caching.
  • Warm‑up overhead is under 1 % of wall‑clock time for 10,000+ batches (under 0.1 % for >100,000 batches).
  • Each cluster mounts its local CNQ instance via NFS and accesses the hub’s data over VPC peering.
  • Full 99 % GPU utilization and p5.48xlarge network saturation with sub‑3 ms data operations on the hub side.
💡 Why It Matters

This architecture eliminates costly data duplication and latency penalties, letting production teams scale training across regions while maintaining near‑optimal GPU utilization and throughput.

✅ Practical Steps

  1. Store the training dataset in a Cloud Native Qumulo (CNQ) instance in a single AWS Region.
  2. In each SageMaker HyperPod cluster, mount the local CNQ instance over NFS and configure VPC peering to the hub CNQ for remote data access.
  3. Rely on Qumulo’s Cloud Data Fabric (CDF) with NeuralCache to automatically cache accessed data on local NVMe, ensuring cross‑Region latency becomes negligible after the initial warm‑up period.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

New in AWS Quick: More capable flows, an updated web experience, and agents that work together

AWS ML Blog•#agents

Three insights you may have missed from theCUBE’s coverage of the Dell Technologies New York Symposium

SiliconANGLE AI•#deployment

Welcome Inkling 2 Mini, Cutting-edge Performance for your Local Setup

Hugging Face Blog•#llm

NVIDIA and Microsoft Expand Accelerated Computing From Azure to Windows PCs

NVIDIA Blog•#compute

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING