Search papers, labs, and topics across Lattice.
This study introduces CWE-Trace, a framework for assessing the ability of large language models (LLMs) to detect vulnerabilities in systems software, utilizing a dataset of 834 curated Linux kernel samples across 74 Common Weakness Enumerations (CWEs). The findings reveal that data contamination does not enhance model performance, with a significant portion of contaminated samples failing to provide useful signals for vulnerability detection, and that fine-tuning does not improve the models' underlying reasoning capabilities. Ultimately, the research highlights a critical disconnect between detection performance and genuine comprehension of security, with the best detection accuracy reaching only 52.1%, underscoring the limitations of current LLMs in this domain.
Calibration without comprehension reveals that fine-tuning LLMs for vulnerability detection fails to enhance their underlying security reasoning, with models achieving only 52.1% detection accuracy.
Whether LLMs scoring well on vulnerability benchmarks genuinely reason about security or merely pattern-match on contaminated data remains unresolved. We present CWE-Trace, a framework for LLM vulnerability detection built from 834 manually curated Linux kernel samples spanning 74 CWEs. The framework enforces a strict temporal split (pre-2025 historical set / post-cutoff leakage-free set), preserves context-aware vulnerable--patched pairs, and introduces two diagnostic metrics: the Directional Failure Index (DFI) and Hierarchical Distance and Direction (HDD). We evaluate eight vanilla LLMs and 15 LoRA fine-tuned variants across non-targeted detection, targeted detection, and CWE classification. Our analysis yields two key results. First, data contamination provides no measurable advantage. Function-level analysis shows that 84% of nominally contaminated samples carry no usable memorization signal: vulnerable functions are absent or cross-mapped across datasets, and ~31% of contaminated samples carry CWE misclassification. Second, backbone directional priors dominate fine-tuning. Models exhibit stable, systematic failure modes (DFI ranging from -85.5 to +94.8 pp) that persist from historical to post-cutoff data and resist correction. Fine-tuning shifts the output threshold without changing the decision policy. This is calibration without comprehension: output distributions adapt to training data while the underlying security reasoning remains absent. The weakest backbone at binary detection (DeepSeek-R1) gains the most in coarse CWE classification, revealing that detection and understanding are decoupled capabilities. The best detection score reaches only 52.1% (+2.1 pp above chance); exact CWE ranking remains below 1.3% Top-1 accuracy, confirming that current LLMs lack reliable security reasoning for systems software, regardless of fine-tuning strategy.
