← Back
AWS ML Blog

How Deepgram optimized voice AI at scale with AWS Trainium

•9 min read•
#compute#deployment
Level:Advanced
For:AI Engineers
✦TL;DR

Deepgram ported its Nova‑3 speech‑to‑text model to AWS Trainium2 using TorchNeuron and achieved an 11‑70 % increase in audio processed per dollar on batch workloads by running eight model replicas on a single trn2.3xlarge EC2 instance. The migration involved compiling the model with torch.compile, eliminating CPU fallbacks, and swapping key layers for fused kernels from the public NKI Library (NKILib). These efficiency gains lower inference costs for Deepgram and its customers, especially those self‑hosting via Amazon SageMaker AI.

⚡ Key Takeaways

  • 11–70 % more audio per dollar processed on Trainium2 versus prior compute options.
  • Used TorchNeuron’s torch.compile to produce static‑shape graphs for encoder and decoder with zero CPU fallbacks.
  • Replaced layers with fused kernels (fused QKV projections, fused FFN, depthwise 1D convolution) from NKILib to boost throughput.
  • Deployed eight Nova‑3 replicas on a single trn2.3xlarge EC2 instance for batch transcription.
  • Numerical accuracy was validated at layer level and end‑to‑end WER to match Deepgram’s reference values.
💡 Why It Matters

Engineers can cut inference spend by over a fifth while maintaining accuracy, enabling larger‑scale voice AI services without expanding hardware budgets.

✅ Practical Steps

  1. Use TorchNeuron to compile the Nova‑3 model with torch.compile in eager mode, ensuring no CPU fallbacks.
  2. Replace target layers with fused kernels from the public NKI Library (NKILib) to improve throughput.
  3. Deploy the optimized model on an Amazon EC2 trn2.3xlarge instance, running eight replicas for batch workloads.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

AWS ML Blog•#deployment

Introducing Unity Gateway Insights

Databricks Blog•#deployment

Introducing Gemini 4 Neon

Google DeepMind•#llm

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

Hugging Face Blog•#llm

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING