Optimizing cost and latency with Amazon Bedrock prompt caching
Prompt caching in Amazon Bedrock can cut input‑token costs by up to 90 percent for repeated context, while also lowering time‑to‑first‑token. The feature works by inserting a **cachePoint** marker in a Converse API request; if the preceding tokens match a cached entry (minimum 1,024 tokens for Claude Sonnet 4.5/4.6, 4,096 tokens for Opus), the model skips reprocessing them. Engineers can apply six ready‑to‑use patterns—including message, system‑prompt, and tool‑definition caching—and control cache lifetimes (default 5 minutes, up to 1 hour) per tenant.
⚡ Key Takeaways
- Cost reduction of up to 90 percent on cached input tokens (e.g., a 10,000‑token contract with 50 questions would otherwise bill 500,000 tokens).
- CachePoint syntax in the Converse API defines the checkpoint; it is model‑agnostic across Anthropic Claude and Amazon Nova families.
- Trade‑off: shorter prompts lower token count but may degrade context quality; caching preserves prompt quality while saving compute.
- Integration: add a **cachePoint** marker and optional **TTL** parameter to your Converse API payload; LangChain users can enable caching via the provided LangChain wrapper.
- Limitation: cache entries are scoped to a single AWS account and Region, and each checkpoint must meet the model‑specific token threshold.
By reusing processed context at the infrastructure layer, engineers can dramatically reduce Bedrock inference spend and latency without altering model prompts, enabling more cost‑effective, responsive AI services.
✅ Practical Steps
- Insert a **cachePoint** marker in your Converse API request payload and set the desired **TTL** (default 5 minutes, up to 1 hour) to enable prompt caching.
- For LangChain workflows, use the LangChain integration example to wrap Bedrock calls with caching enabled.
Want the full story? Read the original article.
Read on AWS ML Blog ↗