Reduce ASR inference costs by 75% with NVIDIA MPS on Amazon EC2
This post shows that using NVIDIA CUDA Multi‑Process Service (MPS) together with the Triton Inference Server on Amazon EC2 cuts the number of required GPUs for Nemotron Parakeet TDT 0.6B V2 ASR serving by 75 % (from 16 L40S GPUs to 4) while keeping sub‑second latency and achieving 92.1 requests per second per GPU. The authors compare three GPU‑sharing mechanisms—time‑slicing, MIG, and MPS—and demonstrate that MPS’s soft SM limits enable four concurrent inference instances on a single L40S, eliminating the 80 % idle SM capacity seen with default time‑slicing. The result is a dramatically lower infrastructure cost for high‑throughput, low‑latency clinical transcription workloads.
⚡ Key Takeaways
- 75 % reduction in GPU instances (16 → 4) while maintaining 92.1 RPS per GPU with sub‑second latency.
- Use NVIDIA CUDA Multi‑Process Service (MPS) to partition an L40S GPU into four concurrent SM slices (≈25 % each).
- Trade‑off: MPS provides soft SM limits and concurrent kernels for many small models, whereas MIG offers hard physical isolation but less flexibility.
- Integrate MPS with NVIDIA Triton Inference Server on Amazon EC2 to schedule requests across the shared GPU slices.
- Prerequisite: Model must be exported to ONNX and optionally optimized with TensorRT to fit within the per‑slice SM budget.
Engineers can dramatically lower cloud GPU spend for latency‑sensitive ASR services by switching from default CUDA time‑slicing to MPS‑enabled Triton serving, enabling higher concurrency on fewer, cheaper instances.
✅ Practical Steps
- Export the fine‑tuned Parakeet TDT 0.6B V2 model to ONNX and apply TensorRT optimizations as described in the prior fine‑tuning post.
- Launch an Amazon EC2 instance with an NVIDIA L40S GPU, install NVIDIA CUDA, enable MPS (`nvidia-cuda-mps-control -d`), and start the Triton Inference Server configured to use MPS.
- Deploy the optimized ONNX/TensorRT model to Triton and verify that each GPU handles ~92 RPS with mean latency < 650 ms (p99 < 1,000 ms).
Want the full story? Read the original article.
Read on AWS ML Blog ↗