CWE-bench v2: More coverage, quality, and transparency
CWE‑bench v2 introduces a public suite of 100 security‑fix tasks covering 91 CWEs, enabling deterministic, programmatic evaluation of AI agents that patch real open‑source repositories. Among 11 evaluated models, Gemini 4 Argon leads with 67.8 % pass@1 and 81.0 % pass@4, closely followed by Claude Opus 5.5 (67.3 % pass@1, 78.0 % pass@4); 95 tasks are solved by at least one model, yet the top model still fails 19 % of the set. Cost‑efficiency analysis shows DeepSeek V4.1 Flash achieving 57.0 % pass@1 at $0.04 per rollout, while Gemini 4 Argon reaches the highest accuracy at $3.92. Engineers can run the benchmark directly via the Harbor Docker harness with a single command.
⚡ Key Takeaways
- 100 tasks span 91 distinct CWEs, providing exhaustive coverage of vulnerability classes.
- Each task is a self‑contained Harbor Docker environment with an instruction and deterministic verifier.
- Pass@1 scores: Gemini 4 Argon 67.8 %, Claude Opus 5.5 67.3 %, DeepSeek V4.1 Flash 57.0 %; pass@4 scores: Gemini 81.0 %, Claude 78.0 %.
- Cost per rollout: DeepSeek $0.04, Claude $1.83, Gemini $3.92, forming a Pareto frontier of performance vs. expense.
- Use the `-k 4` flag to run four attempts per task; the final score is the mean reward over all 400 trials.
The benchmark gives engineers a concrete, reproducible yardstick for measuring how well LLM‑driven agents can autonomously discover and remediate security flaws, directly informing model selection and cost‑benefit decisions for production‑grade cyber‑defense tooling.
✅ Practical Steps
- Clone the CWE‑bench v2 Hugging Face repo and run `harbor -k 4` to execute the full 100‑task suite in Docker.
- Submit your model’s results by opening a PR with a results file to the dataset’s leaderboard repository (or to the Space leaderboard for API‑only models).
- Compare your model’s pass@1 and pass@4 metrics against the published scores to assess both accuracy and cost efficiency.
Want the full story? Read the original article.
Read on Hugging Face Blog ↗