Profiling TorchNeuron (Part 1): System-level profiling on AWS Trainium.
This article demonstrates how to systematically profile inference on AWS Trainium using TorchNeuron, showing how to configure torch.profiler with a NeuronConfig and NeuronProfiler to capture both framework and hardware traces. It explains reading the key_averages() table with correlate_device_time=True to compare Neuron device time against Self CPU time, and using Neuron Explorer to trace an operation from dispatch through runtime scheduling to NeuronCore execution. Engineers gain a reproducible workflow for pinpointing performance bottlenecks on Trainium‑accelerated PyTorch models.
⚡ Key Takeaways
- NeuronConfig parameters such as modes (selecting device instructions and runtime traces) and capture_enabled_for_nc (selecting NeuronCores) control what data is captured.
- NeuronProfiler takes a NeuronConfig and provides an export_trace callback that writes trace.json and the corresponding .neff and .ntff files.
- Setting correlate_device_time=True propagates device time into the key_averages() table, enabling direct CPU vs. Neuron device time comparison.
- Neuron Explorer (accessed via SSH tunnel or VS Code extension) visualizes the system trace, device timeline, and per‑engine breakdown for deeper inspection.
- The output files share a numeric hash linking the compiled executable (.neff) with its hardware instruction trace (.ntff), required for full profile uploads.
By exposing precise CPU and hardware timing for each operator, engineers can iteratively refine model implementations and TorchNeuron configurations to achieve optimal throughput on Trainium, reducing costly trial‑and‑error cycles.
✅ Practical Steps
- Create a NeuronConfig specifying desired modes (e.g., device instructions, runtime traces) and capture_enabled_for_nc to select NeuronCores, then pass it as experimental_config to torch.profiler.profile.
- Use NeuronProfiler with the same NeuronConfig and supply its export_trace callback to write trace.json, .neff, and .ntff files after profiling.
- Open the generated output directory in Neuron Explorer (via SSH tunnel or VS Code extension) to inspect the system trace and correlate it with the key_averages() table.
Want the full story? Read the original article.
Read on Hugging Face Blog ↗