← Back
AWS ML Blog

Building a self-service LLM gateway with Amazon Bedrock and LiteLLM

•18 min read•
#bedrock#llm#deployment
Level:Advanced
For:AI Engineers
✦TL;DR

This article details how to construct a self‑service LLM gateway that fronts Amazon Bedrock models (e.g., Claude Sonnet and Claude Haiku) with LiteLLM running on AWS Fargate, exposing an OpenAI‑compatible API. The pattern leverages Amazon Route 53, CloudFront, an ALB, and per‑user API keys stored in Amazon RDS (PostgreSQL) plus caching in Amazon ElastiCache for Valkey, while CloudWatch monitors RPM and TPM metrics. Engineers can replicate the architecture using the provided AWS CDK snippets to gain fine‑grained cost control, usage observability, and centralized access management.

⚡ Key Takeaways

  • Claude Sonnet and Claude Haiku are used as example foundation models behind the gateway.
  • LiteLLM runs on AWS Fargate with target‑tracking auto‑scaling based on CPU utilization, acting as the OpenAI‑compatible proxy.
  • Per‑user API keys and spend tracking are persisted in Amazon RDS (PostgreSQL) and rate‑limited via Amazon ElastiCache for Valkey.
  • CloudWatch captures requests per minute (RPM) and tokens per minute (TPM) and triggers quota alarms.
  • The gateway is provisioned with AWS CDK constructs that define the Route 53, CloudFront, ALB, Fargate task, IAM task role, and supporting services.
💡 Why It Matters

By abstracting Bedrock behind a single, OpenAI‑compatible endpoint, product teams can rapidly enable AI coding assistants while maintaining strict cost attribution and operational visibility, accelerating deployment cycles in enterprise environments.

✅ Practical Steps

  1. Deploy the LiteLLM Fargate service and supporting resources (RDS, ElastiCache, ALB, etc.) using the provided AWS CDK constructs.
  2. Configure per‑user API keys and spend policies in the LiteLLM self‑service portal, which writes metadata to the Amazon RDS PostgreSQL database.
  3. Set up CloudWatch RPM/TPM alarms to monitor usage and enforce quota limits.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

AWS ML Blog•#deployment

Introducing Unity Gateway Insights

Databricks Blog•#deployment

Introducing Gemini 4 Neon

Google DeepMind•#llm

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

Hugging Face Blog•#llm

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING