← Back
AWS ML Blog

Stop guessing auto scaling targets for TTS on Amazon SageMaker AI

•17 min read•
#deployment#amazon
Level:Advanced
For:AI Engineers
✦TL;DR

This post demonstrates how to eliminate guesswork in SageMaker auto‑scaling for text‑to‑speech by deploying the 82‑million‑parameter Kokoro‑82M model as a custom container and deriving scaling thresholds from two live‑endpoint measurements (Q_max and T_total). By using SageMaker’s native **ConcurrentRequestsPerModel** metric and converting client‑measured concurrency to CloudWatch units (the measured ratio was 1), the auto‑scaling policy accurately reflects the true capacity of a TTS instance. Engineers can now set scale‑out/in thresholds and cooldowns arithmetically, avoiding over‑provisioned GPU costs or hidden latency spikes during traffic bursts.

⚡ Key Takeaways

  • Model: Kokoro‑82M (82 M parameters) deployed on a SageMaker real‑time endpoint.
  • Scaling metric: SageMaker’s native **ConcurrentRequestsPerModel** (Maximum statistic over a 10‑second window).
  • Trade‑off avoided: Prevents idle GPU cost from overly conservative targets and latency breaches from overly aggressive targets.
  • Integration: Auto‑scaling configuration is written using the converted CloudWatch units derived from the measured concurrency ratio (ratio = 1).
  • Prerequisite: Must measure Q_max (max concurrent requests before SLO breach) and T_total (full scale‑out time) on the live endpoint.
💡 Why It Matters

Accurate auto‑scaling eliminates hidden latency failures and unnecessary GPU spend, enabling reliable, cost‑effective TTS services in production.

✅ Practical Steps

  1. Deploy Kokoro‑82M as a custom container on a SageMaker real‑time endpoint using AWS CDK stacks as described in the aws‑samples repository.
  2. Run load tests against the live endpoint to capture Q_max and T_total, then compute the scale‑out/in thresholds and cooldown periods using the provided arithmetic.
  3. Configure the SageMaker auto‑scaling policy to target **ConcurrentRequestsPerModel**, applying the derived CloudWatch units (ratio = 1) to the policy settings.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

AWS ML Blog•#deployment

Introducing Unity Gateway Insights

Databricks Blog•#deployment

Introducing Gemini 4 Neon

Google DeepMind•#llm

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

Hugging Face Blog•#llm

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING