Stop guessing auto scaling targets for TTS on Amazon SageMaker AI
This post demonstrates how to eliminate guesswork in SageMaker auto‑scaling for text‑to‑speech by deploying the 82‑million‑parameter Kokoro‑82M model as a custom container and deriving scaling thresholds from two live‑endpoint measurements (Q_max and T_total). By using SageMaker’s native **ConcurrentRequestsPerModel** metric and converting client‑measured concurrency to CloudWatch units (the measured ratio was 1), the auto‑scaling policy accurately reflects the true capacity of a TTS instance. Engineers can now set scale‑out/in thresholds and cooldowns arithmetically, avoiding over‑provisioned GPU costs or hidden latency spikes during traffic bursts.
⚡ Key Takeaways
- Model: Kokoro‑82M (82 M parameters) deployed on a SageMaker real‑time endpoint.
- Scaling metric: SageMaker’s native **ConcurrentRequestsPerModel** (Maximum statistic over a 10‑second window).
- Trade‑off avoided: Prevents idle GPU cost from overly conservative targets and latency breaches from overly aggressive targets.
- Integration: Auto‑scaling configuration is written using the converted CloudWatch units derived from the measured concurrency ratio (ratio = 1).
- Prerequisite: Must measure Q_max (max concurrent requests before SLO breach) and T_total (full scale‑out time) on the live endpoint.
Accurate auto‑scaling eliminates hidden latency failures and unnecessary GPU spend, enabling reliable, cost‑effective TTS services in production.
✅ Practical Steps
- Deploy Kokoro‑82M as a custom container on a SageMaker real‑time endpoint using AWS CDK stacks as described in the aws‑samples repository.
- Run load tests against the live endpoint to capture Q_max and T_total, then compute the scale‑out/in thresholds and cooldown periods using the provided arithmetic.
- Configure the SageMaker auto‑scaling policy to target **ConcurrentRequestsPerModel**, applying the derived CloudWatch units (ratio = 1) to the policy settings.
Want the full story? Read the original article.
Read on AWS ML Blog ↗