MPS in PyTorch 2.15
PyTorch 2.15 brings FP8 (float8_e4m3fn) support to the MPS backend, cutting tensor memory in half and enabling scaled matrix‑multiply directly on Apple‑silicon GPUs. Mixture‑of‑Experts LLMs now decode 5–6.5× faster—Qwen3.6‑35B‑A3B jumps from 9.4 → 61.3 tokens/s on an M3 Ultra—thanks to native grouped matmul, a dedicated decoding path, and an 8× faster softmax. New Metal kernels also shrink video‑VAE memory from 270 GiB to ~8 GiB and raise Qwen‑Image‑2.1 image‑editing PSNR from 10.06 dB to 29.95 dB. Engineers must run on macOS 15+ with Python 3.11+ to leverage these gains.
⚡ Key Takeaways
- FP8 tensors (float8_e4m3fn) occupy half the space of BF16 and are usable via `F.scaled_mm` on MPS.
- MoE decoding speedup: Qwen3.6‑35B‑A3B improves from 9.4 → 61.3 tokens/s; LFM2.5‑8B‑A1B from 23.6 → 123.6 tokens/s on M3 Ultra with `torch.compile`.
- Single‑row linear layers use specialized matrix‑vector kernels; softmax over vocab is ~8× faster.
- Video VAE memory peak reduced from 270 GiB to ~8 GiB; image‑editing PSNR rises to 29.95 dB after padding fix.
- New public MPS streams and more reliable event timing give Python‑level control over GPU work submission.
The updates let Mac‑based teams run large LLMs and multimodal workloads at production‑grade speed and memory efficiency, removing a major barrier to on‑device AI deployment.
✅ Practical Steps
- Upgrade to macOS 15 or later and install Python 3.11+, then `pip install torch==2.15.*` to get the MPS‑enabled build.
- Compile your MoE model with `torch.compile` on an M3 Ultra to achieve the reported 5–6.5× decoding speedup.
- Switch to FP8 tensors and perform scaled matrix multiplication using `F.scaled_mm` to halve memory usage.
Want the full story? Read the original article.
Read on Hugging Face Blog ↗