← Back
AWS ML Blog

How ArisGlobal scaled generative AI inference on Amazon Bedrock

•11 min read•
#bedrock#llm#enterprise
Level:Advanced
For:AI Engineers
✦TL;DR

ArisGlobal scaled its Nava X generative‑AI inference on Amazon Bedrock by reversing its prompt‑caching strategy—caching the shared prompt prefix instead of the source document—cutting effective input costs despite a 150% growth in shared prompt size and a >200% rise in per‑case inference spend. The system runs Anthropic’s Claude (accessed via Bedrock) and has processed over 1 million regulated transactions, handling billions of tokens across multiple tenants. This caching redesign enabled additional tenants without proportional cost growth, demonstrating a practical cost‑control pattern for multi‑tenant LLM pipelines.

⚡ Key Takeaways

  • Shared prompt prefix grew ~150% over 6 months while per‑case inference cost rose >200%.
  • Caching the shared prompt prefix (instead of the source document) reduced effective input cost.
  • Multi‑tenant inference on Bedrock with Anthropic Claude required EU cross‑Region support and managed prompt caching.
  • Integration uses Amazon Bedrock Guardrails for responsible‑AI filtering of protected health information.
  • Limitation: source documents (10–200 page PDFs) remain uncached and must be sent per request.
💡 Why It Matters

Engineers can dramatically lower inference spend in multi‑tenant LLM services by identifying high‑reuse prompt components and caching them, a tactic that scales cost‑effectively even as prompt size and tenant count grow.

✅ Practical Steps

  1. Configure your Bedrock client to cache the shared prompt prefix (system instructions, extraction schema, few‑shot examples) rather than the case‑specific source document.
  2. Deploy Amazon Bedrock Guardrails alongside your model calls to filter and mask protected health information in generated outputs.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

Introducing Gemini 4 Neon

Google DeepMind•#llm

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

Hugging Face Blog•#llm

NVIDIA RTX Spark Compact Desktops Out Now

NVIDIA Blog•#compute

Helping AI text detectors keep up with changing conditions

MIT News AI•#llm

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING