How Deepgram optimized voice AI at scale with AWS Trainium
Deepgram ported its Nova‑3 speech‑to‑text model to AWS Trainium2 using TorchNeuron and achieved an 11‑70 % increase in audio processed per dollar on batch workloads by running eight model replicas on a single trn2.3xlarge EC2 instance. The migration involved compiling the model with torch.compile, eliminating CPU fallbacks, and swapping key layers for fused kernels from the public NKI Library (NKILib). These efficiency gains lower inference costs for Deepgram and its customers, especially those self‑hosting via Amazon SageMaker AI.
⚡ Key Takeaways
- 11–70 % more audio per dollar processed on Trainium2 versus prior compute options.
- Used TorchNeuron’s torch.compile to produce static‑shape graphs for encoder and decoder with zero CPU fallbacks.
- Replaced layers with fused kernels (fused QKV projections, fused FFN, depthwise 1D convolution) from NKILib to boost throughput.
- Deployed eight Nova‑3 replicas on a single trn2.3xlarge EC2 instance for batch transcription.
- Numerical accuracy was validated at layer level and end‑to‑end WER to match Deepgram’s reference values.
Engineers can cut inference spend by over a fifth while maintaining accuracy, enabling larger‑scale voice AI services without expanding hardware budgets.
✅ Practical Steps
- Use TorchNeuron to compile the Nova‑3 model with torch.compile in eager mode, ensuring no CPU fallbacks.
- Replace target layers with fused kernels from the public NKI Library (NKILib) to improve throughput.
- Deploy the optimized model on an Amazon EC2 trn2.3xlarge instance, running eight replicas for batch workloads.
Want the full story? Read the original article.
Read on AWS ML Blog ↗