How ArisGlobal scaled generative AI inference on Amazon Bedrock
ArisGlobal scaled its Nava X generative‑AI inference on Amazon Bedrock by reversing its prompt‑caching strategy—caching the shared prompt prefix instead of the source document—cutting effective input costs despite a 150% growth in shared prompt size and a >200% rise in per‑case inference spend. The system runs Anthropic’s Claude (accessed via Bedrock) and has processed over 1 million regulated transactions, handling billions of tokens across multiple tenants. This caching redesign enabled additional tenants without proportional cost growth, demonstrating a practical cost‑control pattern for multi‑tenant LLM pipelines.
⚡ Key Takeaways
- Shared prompt prefix grew ~150% over 6 months while per‑case inference cost rose >200%.
- Caching the shared prompt prefix (instead of the source document) reduced effective input cost.
- Multi‑tenant inference on Bedrock with Anthropic Claude required EU cross‑Region support and managed prompt caching.
- Integration uses Amazon Bedrock Guardrails for responsible‑AI filtering of protected health information.
- Limitation: source documents (10–200 page PDFs) remain uncached and must be sent per request.
Engineers can dramatically lower inference spend in multi‑tenant LLM services by identifying high‑reuse prompt components and caching them, a tactic that scales cost‑effectively even as prompt size and tenant count grow.
✅ Practical Steps
- Configure your Bedrock client to cache the shared prompt prefix (system instructions, extraction schema, few‑shot examples) rather than the case‑specific source document.
- Deploy Amazon Bedrock Guardrails alongside your model calls to filter and mask protected health information in generated outputs.
Want the full story? Read the original article.
Read on AWS ML Blog ↗