← Back
Hugging Face Blog

CWE-bench v2: More coverage, quality, and transparency

•5 min read•
#agents#llm
Level:Advanced
For:AI Engineers
✦TL;DR

CWE‑bench v2 introduces a public suite of 100 security‑fix tasks covering 91 CWEs, enabling deterministic, programmatic evaluation of AI agents that patch real open‑source repositories. Among 11 evaluated models, Gemini 4 Argon leads with 67.8 % pass@1 and 81.0 % pass@4, closely followed by Claude Opus 5.5 (67.3 % pass@1, 78.0 % pass@4); 95 tasks are solved by at least one model, yet the top model still fails 19 % of the set. Cost‑efficiency analysis shows DeepSeek V4.1 Flash achieving 57.0 % pass@1 at $0.04 per rollout, while Gemini 4 Argon reaches the highest accuracy at $3.92. Engineers can run the benchmark directly via the Harbor Docker harness with a single command.

⚡ Key Takeaways

  • 100 tasks span 91 distinct CWEs, providing exhaustive coverage of vulnerability classes.
  • Each task is a self‑contained Harbor Docker environment with an instruction and deterministic verifier.
  • Pass@1 scores: Gemini 4 Argon 67.8 %, Claude Opus 5.5 67.3 %, DeepSeek V4.1 Flash 57.0 %; pass@4 scores: Gemini 81.0 %, Claude 78.0 %.
  • Cost per rollout: DeepSeek $0.04, Claude $1.83, Gemini $3.92, forming a Pareto frontier of performance vs. expense.
  • Use the `-k 4` flag to run four attempts per task; the final score is the mean reward over all 400 trials.
💡 Why It Matters

The benchmark gives engineers a concrete, reproducible yardstick for measuring how well LLM‑driven agents can autonomously discover and remediate security flaws, directly informing model selection and cost‑benefit decisions for production‑grade cyber‑defense tooling.

✅ Practical Steps

  1. Clone the CWE‑bench v2 Hugging Face repo and run `harbor -k 4` to execute the full 100‑task suite in Docker.
  2. Submit your model’s results by opening a PR with a results file to the dataset’s leaderboard repository (or to the Space leaderboard for API‑only models).
  3. Compare your model’s pass@1 and pass@4 metrics against the published scores to assess both accuracy and cost efficiency.

Want the full story? Read the original article.

Read on Hugging Face Blog ↗

More like this

Censored vs. Truncated Regression in Python, Explained

Towards Data Science•#llm

New in AWS Quick: More capable flows, an updated web experience, and agents that work together

AWS ML Blog•#agents

How Google Duplex evolved to also call businesses for you in Gemini

Google Research•#agents

Q&A: Enabling more human-like interactions and understanding in multimodal foundation models

MIT News AI•#llm

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING