Building a self-service LLM gateway with Amazon Bedrock and LiteLLM
This article details how to construct a self‑service LLM gateway that fronts Amazon Bedrock models (e.g., Claude Sonnet and Claude Haiku) with LiteLLM running on AWS Fargate, exposing an OpenAI‑compatible API. The pattern leverages Amazon Route 53, CloudFront, an ALB, and per‑user API keys stored in Amazon RDS (PostgreSQL) plus caching in Amazon ElastiCache for Valkey, while CloudWatch monitors RPM and TPM metrics. Engineers can replicate the architecture using the provided AWS CDK snippets to gain fine‑grained cost control, usage observability, and centralized access management.
⚡ Key Takeaways
- Claude Sonnet and Claude Haiku are used as example foundation models behind the gateway.
- LiteLLM runs on AWS Fargate with target‑tracking auto‑scaling based on CPU utilization, acting as the OpenAI‑compatible proxy.
- Per‑user API keys and spend tracking are persisted in Amazon RDS (PostgreSQL) and rate‑limited via Amazon ElastiCache for Valkey.
- CloudWatch captures requests per minute (RPM) and tokens per minute (TPM) and triggers quota alarms.
- The gateway is provisioned with AWS CDK constructs that define the Route 53, CloudFront, ALB, Fargate task, IAM task role, and supporting services.
By abstracting Bedrock behind a single, OpenAI‑compatible endpoint, product teams can rapidly enable AI coding assistants while maintaining strict cost attribution and operational visibility, accelerating deployment cycles in enterprise environments.
✅ Practical Steps
- Deploy the LiteLLM Fargate service and supporting resources (RDS, ElastiCache, ALB, etc.) using the provided AWS CDK constructs.
- Configure per‑user API keys and spend policies in the LiteLLM self‑service portal, which writes metadata to the Amazon RDS PostgreSQL database.
- Set up CloudWatch RPM/TPM alarms to monitor usage and enforce quota limits.
Want the full story? Read the original article.
Read on AWS ML Blog ↗