Stream low-latency speech from AI agents using Kokoro TTS on Amazon SageMaker AI
This post demonstrates how to achieve sub‑second latency for voice agents by streaming each generated sentence to a hosted Kokoro‑82M text‑to‑speech model on Amazon SageMaker AI, allowing audio playback to begin before the full response is produced. Kokoro‑82M (82 M parameters) runs on a single ml.g5.xlarge GPU instance behind a real‑time SageMaker endpoint exposing both a POST /invocations route and a WS /invocations‑bidirectional‑stream route. The solution leverages AWS CDK v2, Python 3.12+, uv, Docker, and Boto3’s invoke_endpoint_with_bidirectional_stream to deliver low‑latency streaming synthesis to browsers via WebSockets. Engineers can now build voice agents that maintain user engagement with near‑real‑time audio output while keeping inference cost low.
⚡ Key Takeaways
- Model: Kokoro‑82M (82 M parameters) deployed on an ml.g5.xlarge GPU instance.
- Endpoint routes: POST /invocations for single‑request synthesis and WS /invocations‑bidirectional‑stream for continuous bidirectional streaming.
- Trade‑off: Small model size enables cheap inference but requires a mid‑tier GPU; streaming reduces perceived latency from ~5 s to <1 s per sentence.
- Integration: Use Boto3’s invoke_endpoint_with_bidirectional_stream API from Python 3.12+ to send sentences as they are generated.
- Prerequisite: Deploy the container with the Docker label com.amazonaws.sagemaker.capabilities.bidirectional-streaming=true and ensure IAM permissions for sagemaker:CreateEndpoint, sagemaker:InvokeEndpointWithBidirectionalStream, and bedrock:InvokeModelWithResponseStream.
By streaming TTS at sentence granularity, production voice agents can deliver responsive user experiences without incurring the high compute cost of larger models, directly impacting engagement metrics and operational spend.
✅ Practical Steps
- Build and push the custom container (Python 3.12, uv, Docker) to Amazon ECR, ensuring the bidirectional‑streaming Docker label is set.
- Deploy the SageMaker AI real‑time endpoint using AWS CDK v2, granting the required IAM actions (sagemaker:CreateEndpoint, sagemaker:InvokeEndpointWithBidirectionalStream, etc.).
- In your agent code, call Boto3’s invoke_endpoint_with_bidirectional_stream to send each generated sentence to the WS /invocations‑bidirectional‑stream route and pipe the returned audio chunks to the browser via a WebSocket connection.
Want the full story? Read the original article.
Read on AWS ML Blog ↗