← Back
Hugging Face Blog

MPS in PyTorch 2.15

•9 min read•
#llm#deployment
Level:Advanced
For:AI Engineers
✦TL;DR

PyTorch 2.15 brings FP8 (float8_e4m3fn) support to the MPS backend, cutting tensor memory in half and enabling scaled matrix‑multiply directly on Apple‑silicon GPUs. Mixture‑of‑Experts LLMs now decode 5–6.5× faster—Qwen3.6‑35B‑A3B jumps from 9.4 → 61.3 tokens/s on an M3 Ultra—thanks to native grouped matmul, a dedicated decoding path, and an 8× faster softmax. New Metal kernels also shrink video‑VAE memory from 270 GiB to ~8 GiB and raise Qwen‑Image‑2.1 image‑editing PSNR from 10.06 dB to 29.95 dB. Engineers must run on macOS 15+ with Python 3.11+ to leverage these gains.

⚡ Key Takeaways

  • FP8 tensors (float8_e4m3fn) occupy half the space of BF16 and are usable via `F.scaled_mm` on MPS.
  • MoE decoding speedup: Qwen3.6‑35B‑A3B improves from 9.4 → 61.3 tokens/s; LFM2.5‑8B‑A1B from 23.6 → 123.6 tokens/s on M3 Ultra with `torch.compile`.
  • Single‑row linear layers use specialized matrix‑vector kernels; softmax over vocab is ~8× faster.
  • Video VAE memory peak reduced from 270 GiB to ~8 GiB; image‑editing PSNR rises to 29.95 dB after padding fix.
  • New public MPS streams and more reliable event timing give Python‑level control over GPU work submission.
💡 Why It Matters

The updates let Mac‑based teams run large LLMs and multimodal workloads at production‑grade speed and memory efficiency, removing a major barrier to on‑device AI deployment.

✅ Practical Steps

  1. Upgrade to macOS 15 or later and install Python 3.11+, then `pip install torch==2.15.*` to get the MPS‑enabled build.
  2. Compile your MoE model with `torch.compile` on an M3 Ultra to achieve the reported 5–6.5× decoding speedup.
  3. Switch to FP8 tensors and perform scaled matrix multiplication using `F.scaled_mm` to halve memory usage.

Want the full story? Read the original article.

Read on Hugging Face Blog ↗

More like this

Censored vs. Truncated Regression in Python, Explained

Towards Data Science•#llm

New in AWS Quick: More capable flows, an updated web experience, and agents that work together

AWS ML Blog•#agents

Three insights you may have missed from theCUBE’s coverage of the Dell Technologies New York Symposium

SiliconANGLE AI•#deployment

How Google Duplex evolved to also call businesses for you in Gemini

Google Research•#agents

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING