Can we change an AI model’s architecture without retraining it from scratch?
The paper “Priming: Hybrid state space models from pre‑trained Transformers” introduces priming, a technique that reuses pretrained Transformer weights to convert them into hybrid architectures with state‑space layers, cutting pre‑training cost by more than 99 %. Experiments show that a 32‑billion‑parameter model using Gated KalmanNet (GKA) matches its original Transformer on reasoning benchmarks while decoding up to 2 × faster and reaching correct answers in as little as 2/3 of the wall‑clock time. This enables engineers to replace attention layers with efficient alternatives without full retraining, dramatically lowering compute expense for long‑context models.
⚡ Key Takeaways
- Pre‑training cost reduction of > 99 % when converting Transformers to hybrid models via priming.
- Gated KalmaNet (GKA) outperforms Gated DeltaNet (GDN) and Mamba‑2, and uniquely supports variable‑length inference without retraining.
- State‑space models consistently beat sliding‑window attention (SWA) even when SWA receives twice the memory budget, delivering the same quality at 2 × throughput.
- Released zoo includes eight‑billion‑parameter and 32‑billion‑parameter primed models, training code, optimized kernels, and a vLLM‑serving plugin under Apache 2.0.
Engineers can now experiment with and deploy long‑context, high‑performance hybrid models at a fraction of the original compute cost, accelerating research cycles and production rollouts.
✅ Practical Steps
- Download the primed hybrid model zoo (8B and 32B) from the authors’ repository.
- Integrate the provided vLLM‑serving plugin to serve the hybrid model in production.
- Use the released training code and optimized kernels to fine‑tune or further adapt the primed models as needed.
Want the full story? Read the original article.
Read on Amazon Science ↗