Use prompt caching with open-weight models on Amazon Bedrock
Amazon Bedrock now lets you reuse long prompt prefixes with open‑weight models via implicit and explicit prompt‑caching modes, cutting input‑token costs and latency. Implicit caching automatically reuses any stable prefix, while explicit caching requires you to mark reusable sections with `prompt_cache_options` set to `explicit` and one‑or‑more `prompt_cache_breakpoint` markers. Cache reads are billed at a lower rate than normal input, and cache writes may be billed higher, so maximizing reuse of static content yields the biggest savings for production agents and coding assistants.
⚡ Key Takeaways
- Implicit caching reuses eligible prefixes automatically on a best‑effort basis; explicit caching uses `prompt_cache_options` = `explicit` plus `prompt_cache_breakpoint` markers.
- You can add several breakpoints (up to the model‑specific maximum) to create layered caches that change at different rates.
- Cache reads are billed at a lower rate than standard input, while cache writes can be billed at a higher rate (see Amazon Bedrock pricing).
- To get the most benefit, place static content (system prompt, reference docs, tool definitions) at the start of the prompt and dynamic content (user query) at the end.
- The cache entry lifetime, minimum prefix length, and maximum breakpoints vary by model; consult the “Models at a glance” table for each model’s limits.
Engineers can halve prompt‑processing latency and reduce token‑based costs for chat‑style or coding agents by structuring prompts to exploit Bedrock’s built‑in caching, enabling cheaper, faster production AI services.
✅ Practical Steps
- In a Chat Completions API call, set `prompt_cache_options` to `explicit` and insert one or more `prompt_cache_breakpoint` markers at the end of each reusable prefix.
- For workloads that already have a stable prefix, simply order the prompt with static content first and rely on implicit caching; no extra parameters are needed.
- Monitor the `cache_usage` fields in each response to verify read/write billing and adjust prefix lengths or breakpoint placement accordingly.
Want the full story? Read the original article.
Read on AWS ML Blog ↗