← Back
AWS ML Blog

Beyond the KV cache: Training and serving i14’s Laplace 3B linear attention model on AWS Trainium2 with native PyTorch

•15 min read•
#llm#deployment#compute
Level:Advanced
For:AI Engineers
✦TL;DR

i14’s Laplace 3B linear‑attention model replaces the KV cache with a fixed 29.5 MB recurrent state, enabling inference that scales independently of context length. Using TorchNeuron Native on an AWS trn2.48xlarge Trainium2 instance, the model trains at about 120,000 tokens / s and serves 11,544 output tokens / s across 1,280 concurrent sequences. The redesign also cuts MuonFSDP2 per‑rank traffic by 24 × and boosts throughput by 1.63 ×, demonstrating a viable path for long‑context LLMs in production.

⚡ Key Takeaways

  • Training throughput: ~120,000 tokens per second on a single trn2.48xlarge Trainium2 instance.
  • Inference throughput: 11,544 output tokens per second across 1,280 concurrent sequences.
  • Architecture change: Replaced the GPU‑specific fused_chunk_simple_gla Triton kernel with standard PyTorch operators using 16 chunks of 128 tokens to match NeuronCore partitions.
  • Muon communication optimization: Two all_to_all collectives in MuonFSDP2 reduced per‑rank traffic by 24 × and increased throughput by 1.63 ×.
  • Integration point: Use TorchNeuron Native (beta) as the PyTorch backend for Trainium2 to run the GLA kernel.
💡 Why It Matters

Engineers can now deploy long‑context LLMs on Trainium2 with predictable memory usage and higher concurrency, reducing serving costs for agentic and document‑analysis workloads.

✅ Practical Steps

  1. Replace the Triton‑based fused_chunk_simple_gla kernel with the provided PyTorch implementation that uses 16 × 128‑token chunks.
  2. Enable TorchNeuron Native (beta) in your PyTorch environment to compile and run the GLA kernel on Trainium2.
  3. Switch Muon to the MuonFSDP2 configuration that employs the two all_to_all collectives for reduced traffic and higher throughput.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

AWS ML Blog•#deployment

Introducing Unity Gateway Insights

Databricks Blog•#deployment

Introducing Gemini 4 Neon

Google DeepMind•#llm

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

Hugging Face Blog•#llm

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING