Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Manual audits reveal that 60% of correct LLM vulnerability verdicts are marred by fabricated or unverifiable claims, with free-form reasoning obscuring these flaws from LLM-as-a-judge evaluations. To address this, we introduce VERA, a framework that requires LLMs to output Structured Reasoning Records (SRRs) detailing tracked pointers, memory operations, and state transitions. VERA's multi-stage judge deterministically audits these SRRs against eight reasoning failure modes, exposing 87% of errors missed by free-form methods.
Most LLM vulnerability reports are right about the bug but wrong about *why*, with free-form reasoning hiding critical flaws that VERA's structured audit uncovers.
Large Language Models (LLMs) are increasingly deployed for automated software vulnerability analysis. Binary classification alone is insufficient; practitioners need explanations to triage bugs and engineer patches. Standard practice relies on Chain-of-Thought (CoT) prompting, but free-form reasoning allows models to obscure logical leaps, hallucinated execution steps, and internal inconsistencies behind plausible prose. Our manual audit reveals that approximately 60% of correct vulnerability verdicts are accompanied by fabricated or unverifiable claims, and free-form explanations allow reasoning errors to evade LLM-as-a-judge evaluation. We present Vulnerability Explanation Reasoning Auditor (VERA), an automated framework for auditing LLM vulnerability reasoning. Rather than accepting free-form text, VERA asks models to output a Structured Reasoning Record (SRR) encoding tracked pointers, memory operations, and state transitions in machine-readable fields. A multi-stage judge audits each SRR against eight reasoning failure modes using deterministic checks, with LLM calls reserved for semantic interpretation. The standardized SRR schema also enables automated mutation testing to benchmark judges at scale without human annotation. Our evaluation shows reasoning flaws occur in correct verdicts just as frequently as incorrect ones, and VERA exposes 87% of reasoning errors that free-form LLM-as-judge systematically miss.