← Back
AWS ML Blog

Stream low-latency speech from AI agents using Kokoro TTS on Amazon SageMaker AI

•14 min read•
#deployment#inference
Level:Advanced
For:AI Engineers
✦TL;DR

This post demonstrates how to achieve sub‑second latency for voice agents by streaming each generated sentence to a hosted Kokoro‑82M text‑to‑speech model on Amazon SageMaker AI, allowing audio playback to begin before the full response is produced. Kokoro‑82M (82 M parameters) runs on a single ml.g5.xlarge GPU instance behind a real‑time SageMaker endpoint exposing both a POST /invocations route and a WS /invocations‑bidirectional‑stream route. The solution leverages AWS CDK v2, Python 3.12+, uv, Docker, and Boto3’s invoke_endpoint_with_bidirectional_stream to deliver low‑latency streaming synthesis to browsers via WebSockets. Engineers can now build voice agents that maintain user engagement with near‑real‑time audio output while keeping inference cost low.

⚡ Key Takeaways

  • Model: Kokoro‑82M (82 M parameters) deployed on an ml.g5.xlarge GPU instance.
  • Endpoint routes: POST /invocations for single‑request synthesis and WS /invocations‑bidirectional‑stream for continuous bidirectional streaming.
  • Trade‑off: Small model size enables cheap inference but requires a mid‑tier GPU; streaming reduces perceived latency from ~5 s to <1 s per sentence.
  • Integration: Use Boto3’s invoke_endpoint_with_bidirectional_stream API from Python 3.12+ to send sentences as they are generated.
  • Prerequisite: Deploy the container with the Docker label com.amazonaws.sagemaker.capabilities.bidirectional-streaming=true and ensure IAM permissions for sagemaker:CreateEndpoint, sagemaker:InvokeEndpointWithBidirectionalStream, and bedrock:InvokeModelWithResponseStream.
💡 Why It Matters

By streaming TTS at sentence granularity, production voice agents can deliver responsive user experiences without incurring the high compute cost of larger models, directly impacting engagement metrics and operational spend.

✅ Practical Steps

  1. Build and push the custom container (Python 3.12, uv, Docker) to Amazon ECR, ensuring the bidirectional‑streaming Docker label is set.
  2. Deploy the SageMaker AI real‑time endpoint using AWS CDK v2, granting the required IAM actions (sagemaker:CreateEndpoint, sagemaker:InvokeEndpointWithBidirectionalStream, etc.).
  3. In your agent code, call Boto3’s invoke_endpoint_with_bidirectional_stream to send each generated sentence to the WS /invocations‑bidirectional‑stream route and pipe the returned audio chunks to the browser via a WebSocket connection.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

AWS ML Blog•#deployment

Introducing Unity Gateway Insights

Databricks Blog•#deployment

Introducing Gemini 4 Neon

Google DeepMind•#llm

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

Hugging Face Blog•#llm

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING