Fix evaluation bias to unlock better models on Amazon Bedrock
Property Finder discovered that 96.8 % of their evaluation labels were inherited from a single incumbent vision model on Amazon Bedrock, causing a systematic bias that made every alternative model appear inferior. By creating an expert‑annotated 256‑image development set and running an iterative prompt‑optimization loop (v0‑v27) scored with Micro F1, they selected a new model that achieved higher accuracy while cutting inference cost by 86 %. This demonstrates that unbiased ground truth and systematic prompt evolution are essential for reliable model selection on Bedrock.
⚡ Key Takeaways
- 96.8 % of original evaluation labels came from the incumbent model’s predictions rather than human verification.
- Prompt‑optimization loop iterated from version v0 to v27, using AI assistants and later automated optimization.
- Micro F1 on a 256‑image expert‑annotated dev set was the metric driving prompt selection.
- The final model delivered higher classification accuracy with 86 % lower cost on Amazon Bedrock.
- Property Finder evaluates 11 quality attributes per image across tens of thousands of daily property photos (≈15 images per listing).
Engineers can avoid costly mis‑selection of models by auditing label sources, building a small human‑verified benchmark, and employing systematic prompt refinement, leading to measurable accuracy gains and substantial cost reductions in production vision pipelines.
✅ Practical Steps
- Assemble a human‑verified development set (e.g., 256 images) covering the target attributes.
- Run candidate models on Bedrock, output predictions in JSON, and compute Micro F1 against the expert set.
- Iterate prompts (v0→v27), using error analysis to adjust wording and leveraging AI assistants or automated tools for refinement.
Want the full story? Read the original article.
Read on AWS ML Blog ↗