← Back
Hugging Face Blog

Welcome Inkling 2 Mini, Cutting-edge Performance for your Local Setup

•12 min read•
#llm#deployment
Welcome Inkling 2 Mini, Cutting-edge Performance for your Local Setup
Level:Advanced
For:AI Engineers
✦TL;DR

Inkling 2 Mini is a 26 B‑parameter multimodal Mixture‑of‑Experts transformer that achieves the capacity of a 48‑layer model by looping 12 middle decoder layers three times, resulting in 3 B active parameters per token and a 1 M‑token context. It routes each token to 8 of 128 experts per layer, supports image and text inputs, and ships in mixed‑precision NVFP4/MXFP8 weights with a DSpark drafter for extra speed. Engineers can run it locally via llama.cpp, MLX, or SGLang, giving performance comparable to Qwen 3.8 27B in preliminary vibe tests.

⚡ Key Takeaways

  • Model size: ~26 B physical parameters, 3 B active parameters per token, 1 M‑token context.
  • Architecture: 24 physical decoder layers with layers 6‑17 looped three times (48 logical layer visits); each layer uses 8‑expert MoE routing from 128 experts.
  • Trade‑off: Looping reuses weights to reduce checkpoint size while maintaining 48 layer visits, incurring extra compute per token but enabling higher capacity on limited hardware.
  • Integration: Use the provided Llama App (llama.cpp), MLX runtime, or SGLang for day‑zero inference on datacenter hardware.
  • Limitation: Hidden size is 2,304 (half of Inkling 2 Small’s 4,608), and the model relies on mixed‑precision NVFP4/MXFP8 weights, requiring compatible hardware.
💡 Why It Matters

Inkling 2 Mini delivers near‑state‑of‑the‑art multimodal performance on a local machine, letting engineers deploy large‑capacity MoE models without the storage overhead of full 48‑layer checkpoints.

✅ Practical Steps

  1. Download the mixed‑precision checkpoint (NVFP4/MXFP8) from Thinking Machines Lab and load it with llama.cpp or MLX as described in the release notes.
  2. Run inference using SGLang’s day‑zero inference API to leverage the DSpark drafter for additional speed‑up.

Want the full story? Read the original article.

Read on Hugging Face Blog ↗

More like this

Censored vs. Truncated Regression in Python, Explained

Towards Data Science•#llm

New in AWS Quick: More capable flows, an updated web experience, and agents that work together

AWS ML Blog•#agents

Three insights you may have missed from theCUBE’s coverage of the Dell Technologies New York Symposium

SiliconANGLE AI•#deployment

How Google Duplex evolved to also call businesses for you in Gemini

Google Research•#agents

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING