← Back
AWS ML Blog

Set up OpenAI ChatGPT Codex with LiteLLM on Amazon ECS and Amazon Bedrock

•18 min read•
#deployment#llm#amazon
Level:Intermediate
For:AI Engineers
✦TL;DR

This article details how to deploy a customer‑operated LiteLLM gateway on Amazon ECS (Fargate) to mediate OpenAI Codex requests to Amazon Bedrock, providing centralized authentication, routing, budgeting, rate‑limiting, and telemetry. The architecture routes Codex’s /v1/responses calls through an Application Load Balancer and AWS WAF before reaching LiteLLM, which then invokes the approved Bedrock model using its ECS task role. Supporting services include Amazon RDS PostgreSQL for state, Secrets Manager/KMS for key storage, and CloudWatch for logging and health monitoring, enabling enterprises to enforce consistent model‑access controls while keeping tool execution local.

⚡ Key Takeaways

  • LiteLLM is deployed on Amazon ECS (Fargate) as a gateway for OpenAI Codex to Amazon Bedrock.
  • An Application Load Balancer and AWS WAF front‑end the gateway, applying network and web‑layer controls.
  • Codex interacts with LiteLLM via the /v1/responses endpoint, which enforces budgets, rate limits, and usage telemetry.
  • The deployment stores state in Amazon RDS PostgreSQL and secrets in AWS Secrets Manager with encryption via AWS KMS.
  • The gateway does not receive a general‑purpose shell and does not replace Codex’s local tool‑approval sandbox.
💡 Why It Matters

Engineers can enforce enterprise‑wide model governance and cost controls without altering developers’ local workflows, simplifying secure, scalable adoption of generative AI coding agents.

✅ Practical Steps

  1. Run the “LiteLLM on AWS quickstart” to provision the LiteLLM gateway on Amazon ECS (Fargate) with the required ALB, WAF, and supporting services.
  2. Configure OpenAI Codex to send its task context and tool definitions to the gateway’s /v1/responses endpoint.
  3. Optionally enable AWS WAF managed protections and source‑IP rate limiting for additional security.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

Censored vs. Truncated Regression in Python, Explained

Towards Data Science•#llm

New in AWS Quick: More capable flows, an updated web experience, and agents that work together

AWS ML Blog•#agents

Three insights you may have missed from theCUBE’s coverage of the Dell Technologies New York Symposium

SiliconANGLE AI•#deployment

How Google Duplex evolved to also call businesses for you in Gemini

Google Research•#agents

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING