I wrote CVE-Bench two months ago: an open source benchmark with 20 real-world security vulnerabilities designed to test AI agents' ability to discover and repair security vulnerabilities. I'm expanding it by 10 more tasks and I'd like the community's input on what I'm building. Which CWE categories or specific CVEs should I prioritize?
I wrote CVE-Bench two months ago: an open source benchmark with 20 real-world security vulnerabilities designed to test AI agents' ability to discover and repair security vulnerabilities. I'm expanding it by 10 more tasks and I'd like the community's input on what I'm building. Which CWE categories or specific CVEs should I prioritize?
===
Current CVEs are
CVE-2025-64512 (pdfminer.six)
CVE-2026-22250 (WeblateOrg/wlc)
CVE-2026-24486 (python-multipart)
CVE-2026-26331 (yt-dlp)
CVE-2026-30930 (glances)
CVE-2026-32714 (scitokens)
CVE-2026-33175 (jupyterhub/oauthenticator)
CVE-2026-33545 (MobSF)
CVE-2026-35171 (kedro)
CVE-2026-40864 (jupyterhub)
CVE-2026-42215 (GitPython)
CVE-2026-42310 (Pillow)
CVE-2026-42351 (pygeoapi)
CVE-2026-42561 (python-multipart)
CVE-2026-44307 (mako)
CVE-2026-44431 (urllib3)
CVE-2026-44432 (urllib3)
CVE-2026-44502 (bugsink)
CVE-2026-45017 (liquid)
GHSA-r758-8hxw-4845 (justhtml)
===
CWE are
CWE-22 (Path Traversal): 4
CWE-89 (SQL Injection): 3
CWE-502 (Deserialization): 2
CWE-78 (OS Command Injection): 2
CWE-287 / CWE-290 (Auth bypass): 1 each
CWE-295, CWE-94, CWE-352, CWE-835, CWE-770, CWE-200, CWE-409, CWE-918, CWE-79: 1 each
===
I know it is a bit concentrated and I'd like to bring in more relevant cases.
I'd like to have recommendations on CWEs you'd like to see in CVE-Bench or even specific CVEs you'd like added.
If you're providing specific CVEs, I have a few requirements
* Must be a pure-Python project (otherwise I'll have trouble setting up the running environment);
* Preferably a known real-world project or library;
* Local project setup should be relatively simple, not a 4h nightmare.
* The CVE description should be reasonably well documented.
Which CWEs am I missing, or what specific CVE would you add, given those constraints?
Thanks for the contribution