Introducing usage-based pricing for Amazon SageMaker AI bidirectional streaming
Amazon SageMaker AI now supports usage‑based metering for bidirectional streaming inference, enabling model providers to bill AWS Marketplace customers based on actual streamed usage rather than static endpoint time. The feature adds a companion WebSocket metadata stream (opened at /invocations‑bidirectional‑stream‑metadata) that carries JSON metering records, and model servers signal support with the X‑Amzn‑SageMaker‑Metadata‑Stream‑Supported: true header. Deepgram is the launch partner, bringing real‑time Speech‑to‑Text and Text‑to‑Speech billing to the marketplace. Engineers can integrate this metering without changing existing data streams, aligning costs with real‑time AI consumption.
⚡ Key Takeaways
- Deepgram is the launch partner for usage‑based billing of real‑time STT and TTS on SageMaker AI.
- Model servers must return the header X‑Amzn‑SageMaker‑Metadata‑Stream‑Supported: true to opt‑in to the metadata stream.
- A second WebSocket connection to /invocations‑bidirectional‑stream‑metadata carries JSON metering frames alongside the primary inference stream.
- SageMaker AI endpoint infrastructure charges continue to apply separately from the usage‑based software charge.
- Billing aligns with actual streamed usage, avoiding charges during idle periods of long‑lived streams.
🔧 Tools & Libraries
Aligning software charges with real‑time streaming consumption lets enterprises pay only for the AI work performed, reducing waste and simplifying cost forecasting for production‑grade, low‑latency AI services.
✅ Practical Steps
- Ensure your model server returns X‑Amzn‑SageMaker‑Metadata‑Stream‑Supported: true in the WebSocket upgrade response to enable the companion metadata stream.
- Implement a WebSocket client that connects to /invocations‑bidirectional‑stream‑metadata and sends metering data as JSON text frames, using the same X‑Amzn‑SageMaker‑Request‑Id as the inference stream for correlation.
- Deploy the model on SageMaker AI as usual; the platform will automatically open the metadata WebSocket alongside the inference stream for usage‑based metering.
Want the full story? Read the original article.
Read on AWS ML Blog ↗