Welcome Inkling 2 Mini, Cutting-edge Performance for your Local Setup
Inkling 2 Mini is a 26 B‑parameter multimodal Mixture‑of‑Experts transformer that achieves the capacity of a 48‑layer model by looping 12 middle decoder layers three times, resulting in 3 B active parameters per token and a 1 M‑token context. It routes each token to 8 of 128 experts per layer, supports image and text inputs, and ships in mixed‑precision NVFP4/MXFP8 weights with a DSpark drafter for extra speed. Engineers can run it locally via llama.cpp, MLX, or SGLang, giving performance comparable to Qwen 3.8 27B in preliminary vibe tests.
⚡ Key Takeaways
- Model size: ~26 B physical parameters, 3 B active parameters per token, 1 M‑token context.
- Architecture: 24 physical decoder layers with layers 6‑17 looped three times (48 logical layer visits); each layer uses 8‑expert MoE routing from 128 experts.
- Trade‑off: Looping reuses weights to reduce checkpoint size while maintaining 48 layer visits, incurring extra compute per token but enabling higher capacity on limited hardware.
- Integration: Use the provided Llama App (llama.cpp), MLX runtime, or SGLang for day‑zero inference on datacenter hardware.
- Limitation: Hidden size is 2,304 (half of Inkling 2 Small’s 4,608), and the model relies on mixed‑precision NVFP4/MXFP8 weights, requiring compatible hardware.
Inkling 2 Mini delivers near‑state‑of‑the‑art multimodal performance on a local machine, letting engineers deploy large‑capacity MoE models without the storage overhead of full 48‑layer checkpoints.
✅ Practical Steps
- Download the mixed‑precision checkpoint (NVFP4/MXFP8) from Thinking Machines Lab and load it with llama.cpp or MLX as described in the release notes.
- Run inference using SGLang’s day‑zero inference API to leverage the DSpark drafter for additional speed‑up.
Want the full story? Read the original article.
Read on Hugging Face Blog ↗