Prepare web content for Amazon Bedrock Knowledge Bases with Crawl4AI
Amazon Bedrock Knowledge Bases provides a fully managed RAG pipeline that can ingest web content via the built‑in Web Crawler connector, but for sites that require content pruning, targeted extraction, or custom metadata you can prepend a preparation step using the open‑source Crawl4AI Python crawler, which writes Markdown and side‑car metadata to Amazon S3 for ingestion. The solution supports both managed and customer‑managed knowledge bases, runs the Crawl4AI workflow in an AWS Fargate task, and leverages default components like DefaultMarkdownGenerator with a pruning filter and optional CSS selectors for region‑specific extraction. Engineers can choose the Web Crawler path when it meets their needs or switch to the Crawl4AI path for finer‑grained control, then validate retrieval quality after ingestion.
⚡ Key Takeaways
- The Crawl4AI step outputs Markdown and a metadata side‑car per page to Amazon S3 for Bedrock Knowledge Base ingestion.
- Use DefaultMarkdownGenerator with a pruning content filter to remove navigation/boilerplate while preserving headings, lists, and tables.
- Custom metadata (source URL, crawl timestamp, content type, user‑defined attributes) can be attached to each document for filtered retrieval.
- The workflow runs as an AWS Fargate task in a public subnet, allowing the crawler to reach configured websites.
- Managed and customer‑managed knowledge bases both accept Amazon S3 data sources, so the Crawl4AI path works with either type.
By integrating Crawl4AI you gain deterministic control over what web content enters your Bedrock Knowledge Base, improving retrieval relevance and enabling metadata‑driven filtering without managing a vector store yourself.
✅ Practical Steps
- Clone the companion aws-samples repository and run the provided AWS Fargate task that executes Crawl4AI, writing Markdown and metadata side‑cars to an S3 bucket.
- Configure Crawl4AI’s DefaultMarkdownGenerator with a pruning filter and, if needed, a CSS selector to extract only the desired page region before ingestion.
- Point the Bedrock Knowledge Base’s Amazon S3 data source to the bucket containing the prepared files and trigger ingestion, then run retrieval queries to validate quality.
Want the full story? Read the original article.
Read on AWS ML Blog ↗