← Back
AWS ML Blog

Amazon Bedrock tokenomics data lake

•11 min read•
#bedrock#deployment
Level:Intermediate
For:AI Engineers
✦TL;DR

Amazon Bedrock now has a centralized observability data lake that aggregates raw invocation logs from linked accounts into a single, queryable Athena table. The solution uses CloudFormation‑deployed Lambda, CloudWatch subscription filters, and Kinesis Data Firehose to flatten prompts, responses, token counts, model IDs, stop reasons, timestamps, and IAM roles into a unified view. Engineers can instantly query per‑invocation costs and debug usage across the organization.

⚡ Key Takeaways

  • The processed Athena table includes prompt text, model response, input and output token counts, model name, stop reason, account ID, timestamp, and IAM role per invocation.
  • Architecture relies on CloudWatch subscription filters forwarding logs to a central account, then Kinesis Data Firehose delivering to S3 for Athena querying.
  • Centralized view eliminates the need to manually aggregate logs across accounts, reducing debugging latency and spotting wasteful prompts.
  • Deploy the solution with two AWS CloudFormation templates provided in the AWS Sample Bedrock Invocation Data Lake GitHub repo.
  • Requires enabling Bedrock model invocation logging in each source AWS account and linking those accounts to the central account.
💡 Why It Matters

Engineers can now attribute token usage and costs to specific prompts and teams in real time, enabling precise optimization and billing accountability for large‑scale Bedrock deployments.

✅ Practical Steps

  1. Enable Bedrock model invocation logging in each AWS account’s Bedrock settings.
  2. Deploy the two CloudFormation templates from the AWS Sample Bedrock Invocation Data Lake GitHub repository to set up the Lambda, CloudWatch subscription filters, and Kinesis Data Firehose pipeline.
  3. Query the consolidated Athena table to retrieve per‑invocation details such as prompt, token counts, and cost.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

AWS ML Blog•#deployment

Introducing Unity Gateway Insights

Databricks Blog•#deployment

Introducing Gemini 4 Neon

Google DeepMind•#llm

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

Hugging Face Blog•#llm

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING