← Back
AWS ML Blog

Reduce ASR inference costs by 75% with NVIDIA MPS on Amazon EC2

•16 min read•
#deployment#inference#nvidia
Level:Advanced
For:AI Engineers
✦TL;DR

This post shows that using NVIDIA CUDA Multi‑Process Service (MPS) together with the Triton Inference Server on Amazon EC2 cuts the number of required GPUs for Nemotron Parakeet TDT 0.6B V2 ASR serving by 75 % (from 16 L40S GPUs to 4) while keeping sub‑second latency and achieving 92.1 requests per second per GPU. The authors compare three GPU‑sharing mechanisms—time‑slicing, MIG, and MPS—and demonstrate that MPS’s soft SM limits enable four concurrent inference instances on a single L40S, eliminating the 80 % idle SM capacity seen with default time‑slicing. The result is a dramatically lower infrastructure cost for high‑throughput, low‑latency clinical transcription workloads.

⚡ Key Takeaways

  • 75 % reduction in GPU instances (16 → 4) while maintaining 92.1 RPS per GPU with sub‑second latency.
  • Use NVIDIA CUDA Multi‑Process Service (MPS) to partition an L40S GPU into four concurrent SM slices (≈25 % each).
  • Trade‑off: MPS provides soft SM limits and concurrent kernels for many small models, whereas MIG offers hard physical isolation but less flexibility.
  • Integrate MPS with NVIDIA Triton Inference Server on Amazon EC2 to schedule requests across the shared GPU slices.
  • Prerequisite: Model must be exported to ONNX and optionally optimized with TensorRT to fit within the per‑slice SM budget.
💡 Why It Matters

Engineers can dramatically lower cloud GPU spend for latency‑sensitive ASR services by switching from default CUDA time‑slicing to MPS‑enabled Triton serving, enabling higher concurrency on fewer, cheaper instances.

✅ Practical Steps

  1. Export the fine‑tuned Parakeet TDT 0.6B V2 model to ONNX and apply TensorRT optimizations as described in the prior fine‑tuning post.
  2. Launch an Amazon EC2 instance with an NVIDIA L40S GPU, install NVIDIA CUDA, enable MPS (`nvidia-cuda-mps-control -d`), and start the Triton Inference Server configured to use MPS.
  3. Deploy the optimized ONNX/TensorRT model to Triton and verify that each GPU handles ~92 RPS with mean latency < 650 ms (p99 < 1,000 ms).

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

AWS ML Blog•#deployment

Introducing Unity Gateway Insights

Databricks Blog•#deployment

Introducing Gemini 4 Neon

Google DeepMind•#llm

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

Hugging Face Blog•#llm

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING