Mixture-of-Experts kernel engineering on Amazon SageMaker AI
This article details kernel‑level optimizations for fused Mixture‑of‑Experts (MoE) models on Amazon SageMaker AI, focusing on leveraging NVIDIA Hopper’s Warp Group Matrix Multiply‑Accumulate (WGMMA) instructions. By restructuring the expert computation to satisfy Hopper’s fixed‑size M dimension (M = 64) and merging routing, shuffling, and expert calculations into a single GPU kernel, the authors achieve higher tensor‑core utilization and reduced latency for thin‑batch decode phases (e.g., M = 1). Engineers can apply these fused‑MoE kernels to improve per‑token compute efficiency and lower inference cost on SageMaker.
⚡ Key Takeaways
- The Hopper WGMMA ISA requires the leading matrix dimension M to be ≥ 64, otherwise compute lanes idle.
- Fused MoE merges token routing, data shuffling, and expert matrix multiplications into one batched GPU operation.
- Decode‑phase token rows often drop below the hardware‑optimal M = 64 (e.g., M = 1), creating a performance bottleneck.
- The optimization swaps operand roles (Swap‑AB) so that the larger dimension maps to Operand B, aligning with WGMMA’s variable‑N slot.
- Applying the fused kernel on SageMaker reduces latency by avoiding separate sequential loops for routing and expert computation.
🔧 Tools & Libraries
By aligning MoE tensor shapes with Hopper’s WGMMA constraints, engineers can cut inference latency and cloud compute costs for large‑scale LLM services deployed on SageMaker.
✅ Practical Steps
- Integrate the provided fused MoE kernel (which implements Swap‑AB for WGMMA) into your SageMaker inference container.
- Ensure that expert weight matrices are shaped so that the token‑row dimension (M) meets or exceeds 64, or rely on the kernel’s operand swap to satisfy this requirement.
- Deploy the updated container to SageMaker endpoints and benchmark latency against the prior sequential routing implementation.
Want the full story? Read the original article.
Read on AWS ML Blog ↗