Your Experts Have Partners. There's HOPE for Keeping Them Together
Higher-Order Pruning of Experts (HOPE) introduces a second‑order pruning method that accounts for pairwise interactions between experts in Mixture‑of‑Experts (MoE) LLMs. At 50 % pruning, HOPE attains the best average rank among competitors and yields up to +6.1 % improvement on agentic coding tasks compared to the next‑best method. By permanently deleting the least useful experts without additional training or latency‑inducing dynamic loading, HOPE reduces the massive memory overhead of MoEs (e.g., 95 % of Qwen3.5‑122B’s ~250 GB footprint resides in idle expert MLPs). Engineers can apply HOPE to shrink MoE models while preserving performance.
⚡ Key Takeaways
- 50 % pruning with HOPE delivers up to +6.1 % gain on agentic coding benchmarks over the runner‑up pruning technique.
- HOPE incorporates pairwise expert interaction scores rather than independent scalar importance scores.
- The method avoids latency penalties of dynamic expert loading and works alongside quantization.
- It is applicable to any MoE layer where each token routes to K = 8 experts out of E (e.g., 256 experts per layer in Qwen3.5‑122B).
- Pruning is performed without additional training, requiring only a calibration dataset for scoring.
HOPE lets production teams cut MoE memory consumption by half while actually improving task‑specific accuracy, enabling deployment of larger LLMs on hardware with tighter memory budgets.
✅ Practical Steps
- Apply the HOPE pruning pipeline described in the paper to your MoE model, using a calibration dataset to compute pairwise expert interaction scores.
Want the full story? Read the original article.
Read on Hugging Face Blog ↗