← Back
AWS ML Blog

Prepare web content for Amazon Bedrock Knowledge Bases with Crawl4AI

•22 min read•
#rag#bedrock#deployment
Level:Advanced
For:AI Engineers
✦TL;DR

Amazon Bedrock Knowledge Bases provides a fully managed RAG pipeline that can ingest web content via the built‑in Web Crawler connector, but for sites that require content pruning, targeted extraction, or custom metadata you can prepend a preparation step using the open‑source Crawl4AI Python crawler, which writes Markdown and side‑car metadata to Amazon S3 for ingestion. The solution supports both managed and customer‑managed knowledge bases, runs the Crawl4AI workflow in an AWS Fargate task, and leverages default components like DefaultMarkdownGenerator with a pruning filter and optional CSS selectors for region‑specific extraction. Engineers can choose the Web Crawler path when it meets their needs or switch to the Crawl4AI path for finer‑grained control, then validate retrieval quality after ingestion.

⚡ Key Takeaways

  • The Crawl4AI step outputs Markdown and a metadata side‑car per page to Amazon S3 for Bedrock Knowledge Base ingestion.
  • Use DefaultMarkdownGenerator with a pruning content filter to remove navigation/boilerplate while preserving headings, lists, and tables.
  • Custom metadata (source URL, crawl timestamp, content type, user‑defined attributes) can be attached to each document for filtered retrieval.
  • The workflow runs as an AWS Fargate task in a public subnet, allowing the crawler to reach configured websites.
  • Managed and customer‑managed knowledge bases both accept Amazon S3 data sources, so the Crawl4AI path works with either type.
💡 Why It Matters

By integrating Crawl4AI you gain deterministic control over what web content enters your Bedrock Knowledge Base, improving retrieval relevance and enabling metadata‑driven filtering without managing a vector store yourself.

✅ Practical Steps

  1. Clone the companion aws-samples repository and run the provided AWS Fargate task that executes Crawl4AI, writing Markdown and metadata side‑cars to an S3 bucket.
  2. Configure Crawl4AI’s DefaultMarkdownGenerator with a pruning filter and, if needed, a CSS selector to extract only the desired page region before ingestion.
  3. Point the Bedrock Knowledge Base’s Amazon S3 data source to the bucket containing the prepared files and trigger ingestion, then run retrieval queries to validate quality.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

AWS ML Blog•#deployment

Introducing Unity Gateway Insights

Databricks Blog•#deployment

Introducing Gemini 4 Neon

Google DeepMind•#llm

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

Hugging Face Blog•#llm

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING