← Back
Hugging Face Blog

180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers

•10 min read•
#inference#llm
Level:Advanced
For:AI Engineers
✦TL;DR

The article demonstrates that the Strata inference engine can run a 180 B parameter Mixture‑of‑Experts (MoE) model by streaming experts from SSD, achieving token‑generation quality within 6 % of the on‑GPU Strata baseline. Compared to llama.cpp, vLLM, and Hugging Face Transformers, Strata keeps only hot experts in VRAM while offloading the rest to RAM or disk, dramatically reducing GPU memory requirements. Benchmarks show Qwen‑3.8‑Flash‑Next running on a 32 GB Mac M4 at 13.5 tok/s, and off‑loading only experts in vLLM doubles its speed with additional FP8 gains. Engineers can thus deploy very large MoE models on commodity hardware by leveraging tiered expert storage.

⚡ Key Takeaways

  • 180 B‑parameter MoE model runs with token‑generation quality within 6 % of Strata’s full‑GPU baseline by streaming experts from SSD.
  • Strata’s design keeps hot experts in VRAM and moves cold experts to RAM or disk, unlike other runtimes that keep all experts on CPU or repeatedly transfer them over PCIe.
  • Off‑loading only experts in vLLM (using `--cpu-offload-params experts`) doubles inference speed; adding FP8 quantization further reduces memory bandwidth.
  • llama.cpp supports MoE off‑load via flags such as `--cpu-moe`, `--n-cpu-moe N`, `-ot "exps=CPU"` and a newly added GPU expert cache (`--moe-cache-mib`).
  • Transformers’ Accelerate off‑load (`device_map="auto", offload_folder`) copies whole expert banks per token, offering no caching and no memory savings on unified‑memory systems.
💡 Why It Matters

By tiering expert storage, engineers can run state‑of‑the‑art MoE models on a single consumer GPU or even a laptop, unlocking high‑capacity LLM capabilities without costly multi‑GPU clusters.

✅ Practical Steps

  1. Use Strata to run a 180 B MoE model, ensuring hot experts fit in VRAM and configuring SSD tiering for cold experts.
  2. For llama.cpp, enable CPU‑only MoE with `--cpu-moe` (or `--n-cpu-moe N`) and optionally activate the GPU expert cache via `--moe-cache-mib`.
  3. In vLLM, accelerate inference by setting `--cpu-offload-params experts` and enable FP8 quantization to halve expert memory bandwidth.

Want the full story? Read the original article.

Read on Hugging Face Blog ↗

More like this

Censored vs. Truncated Regression in Python, Explained

Towards Data Science•#llm

How Google Duplex evolved to also call businesses for you in Gemini

Google Research•#agents

Q&A: Enabling more human-like interactions and understanding in multimodal foundation models

MIT News AI•#llm

Welcome Inkling 2 Mini, Cutting-edge Performance for your Local Setup

Hugging Face Blog•#llm

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING