180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers
The article demonstrates that the Strata inference engine can run a 180 B parameter Mixture‑of‑Experts (MoE) model by streaming experts from SSD, achieving token‑generation quality within 6 % of the on‑GPU Strata baseline. Compared to llama.cpp, vLLM, and Hugging Face Transformers, Strata keeps only hot experts in VRAM while offloading the rest to RAM or disk, dramatically reducing GPU memory requirements. Benchmarks show Qwen‑3.8‑Flash‑Next running on a 32 GB Mac M4 at 13.5 tok/s, and off‑loading only experts in vLLM doubles its speed with additional FP8 gains. Engineers can thus deploy very large MoE models on commodity hardware by leveraging tiered expert storage.
⚡ Key Takeaways
- 180 B‑parameter MoE model runs with token‑generation quality within 6 % of Strata’s full‑GPU baseline by streaming experts from SSD.
- Strata’s design keeps hot experts in VRAM and moves cold experts to RAM or disk, unlike other runtimes that keep all experts on CPU or repeatedly transfer them over PCIe.
- Off‑loading only experts in vLLM (using `--cpu-offload-params experts`) doubles inference speed; adding FP8 quantization further reduces memory bandwidth.
- llama.cpp supports MoE off‑load via flags such as `--cpu-moe`, `--n-cpu-moe N`, `-ot "exps=CPU"` and a newly added GPU expert cache (`--moe-cache-mib`).
- Transformers’ Accelerate off‑load (`device_map="auto", offload_folder`) copies whole expert banks per token, offering no caching and no memory savings on unified‑memory systems.
By tiering expert storage, engineers can run state‑of‑the‑art MoE models on a single consumer GPU or even a laptop, unlocking high‑capacity LLM capabilities without costly multi‑GPU clusters.
✅ Practical Steps
- Use Strata to run a 180 B MoE model, ensuring hot experts fit in VRAM and configuring SSD tiering for cold experts.
- For llama.cpp, enable CPU‑only MoE with `--cpu-moe` (or `--n-cpu-moe N`) and optionally activate the GPU expert cache via `--moe-cache-mib`.
- In vLLM, accelerate inference by setting `--cpu-offload-params experts` and enable FP8 quantization to halve expert memory bandwidth.
Want the full story? Read the original article.
Read on Hugging Face Blog ↗