Reduce RAG costs on Amazon Bedrock with query-aware compression
This article presents a query‑aware compression pattern for Retrieval‑Augmented Generation on Amazon Bedrock that filters retrieved chunks with a low‑cost model (Anthropic Claude Haiku) before passing only the relevant spans to a higher‑capacity model (Anthropic Claude Sonnet). By inserting this post‑retrieval step in an AWS Lambda function, the number of input tokens sent to the primary model drops dramatically, cutting inference cost while preserving answer quality and reducing hallucination risk. The approach can be combined with Bedrock features such as prompt caching, Intelligent Prompt Routing, and the Rerank API for additional savings.
⚡ Key Takeaways
- Uses Claude Haiku as the compression model and Claude Sonnet as the primary model in the token‑reduction cascade.
- Implements the filtering step as a custom post‑retrieval processor in an AWS Lambda function.
- Reduces the per‑query token count from several thousand (typical 5–20 chunks) to only the verbatim relevant spans, lowering Bedrock inference cost.
- Integrates with existing Bedrock capabilities: prompt caching, Intelligent Prompt Routing, and the Rerank API to compound savings.
- Requires an IAM role with Bedrock model access permissions for both Claude Haiku and Claude Sonnet.
Engineers can achieve measurable cost reductions on large‑scale RAG deployments without sacrificing answer fidelity, enabling more economical production of knowledge‑base assistants on Bedrock.
✅ Practical Steps
- Deploy the provided AWS Lambda function that calls Claude Haiku to filter retrieved chunks before invoking Claude Sonnet.
- Grant the Lambda’s IAM role Bedrock model access for both Claude Haiku and Claude Sonnet as described in the “Create a role to delegate permissions to an AWS service” guide.
- Enable Bedrock prompt caching or Intelligent Prompt Routing alongside the Lambda to further reduce token usage and latency.
Want the full story? Read the original article.
Read on AWS ML Blog ↗