Does Your Model Run on AWS Trainium? Look It Up
The new torch‑neuronx Space lets engineers instantly verify whether any Hugging Face Transformer model runs unmodified on AWS Trainium 2, showing sharding strategy, wall‑time (including compilation) and peak memory. Results are drawn from the public aws‑neuron/torch‑neuronx‑model‑support dataset, which currently covers only text‑generation and image‑text‑to‑text models that fit on up to 16 Trainium cores (four trn2 instances). The Space also provides the exact run.py script, required package list, and a command to preload compiled NEFF graphs to skip the one‑time compilation step. Engineers can thus quickly prototype and benchmark Trainium deployments without manual trial‑and‑error.
⚡ Key Takeaways
- Only models fitting on 4 Trainium 2 devices (≤ 16 cores) are tested; a PASS does not guarantee throughput or long‑generation performance.
- Models are sharded using tensor parallelism; Mixture‑of‑Experts models use expert parallelism when no tensor‑parallel plan exists.
- The Space displays the Neuron stack version (neuronx‑cc) and Transformers version used for each test.
- A command is provided to copy pre‑compiled NEFF graphs from the public cache, eliminating the initial torch.compile compilation step.
- The underlying dataset is a Parquet file per neuronx‑cc/library version pair, accessible via the aws‑neuron/torch‑neuronx‑model‑support dataset.
Engineers can now determine Trainium compatibility and obtain ready‑to‑run scripts in seconds, dramatically reducing the time to prototype high‑performance inference on AWS’s custom silicon.
✅ Practical Steps
- Open the torch‑neuronx Space, select the Selected view (or Trending/Downloads view) and type the model name to see compatibility, sharding details, wall‑time and memory.
- Install the listed packages on a Trainium 2 instance (e.g., via the AWS Neuron Deep Learning Container) and run the provided run.py script, which handles its own torch.distributed launch.
- To avoid recompilation, execute the Space‑provided command that copies the pre‑built NEFF graphs from the public cache using the matching neuronx‑cc version.
Want the full story? Read the original article.
Read on Hugging Face Blog ↗