Search papers, labs, and topics across Lattice.
This study introduces HoF-Bench, a benchmark created from 95 AI-discovered CVEs across eight repositories, to evaluate the effectiveness of LLM-based analyzers in identifying vulnerabilities. The research reveals that a minimal LLM-based analyzer successfully rediscovers 68% of these CVEs under strict conditions, while no frontier models were able to perform detection. The findings highlight significant challenges in detecting vulnerabilities in C infrastructure code, emphasizing the need for improved methodologies in vulnerability scanning.
A minimal LLM-based analyzer can rediscover 68% of AI-discovered CVEs, while frontier models fail to detect any, exposing critical gaps in current vulnerability detection methods.
LLM-based analyzers have begun finding real vulnerabilities in mature open-source projects: AISLE's analyzer is credited with more than 280 CVEs across 78 projects, including OpenSSL, curl, and GnuTLS. We introduce HoF-Bench (named after AISLE's public Hall of Fame), a benchmark built from 95 of these public AI-discovered CVEs across eight repositories pinned at vulnerable commits. Analyzers receive source and target-file scope but not CVE identifiers, descriptions, fixes, or expected mechanisms; a detector-blinded frontier-model judge credits only findings that identify the same code path, root cause, attack condition, and impact. A deliberately minimal LLM-based analyzer rediscovers up to 65 of the 95 CVEs (68%) under this strict protocol. No frontier model performs detection anywhere in the study. The ten detector backbones are five open-weight models (21B--284B total parameters, 3--13B active) and five proprietary small or"flash"-tier models. All of them run in the fixed scaffold with four repeated passes, an optional generated-context stage, and a replayable multi-round triage stage (7,600 model--CVE pass records). Difficulty is strongly structured by language; the CVEs missed by every model concentrate in C infrastructure code. HoF-Bench provides a compact test bed for comparing vulnerability scanners, their reliability across repeated runs, and the candidate volume they create. The dataset is available at https://huggingface.co/datasets/aisleinc/HoF-Bench.
