Search papers, labs, and topics across Lattice.
RustMizan is a novel benchmarking framework designed for analyzing Rust vulnerabilities, addressing limitations of existing benchmarks that use non-compilable snippets and binary classification. It incorporates compilable code variants with detailed annotations for vulnerability detection, classification, and localization, while also employing a mutation framework for contamination testing. The framework reveals that while binary classification accuracy is relatively moderate (56-65%), line localization performance is significantly lower, highlighting challenges in robustness against adversarial cues which reduce line F1 scores by approximately 27%.
RustMizan exposes critical weaknesses in vulnerability detection, revealing that even advanced models struggle with line localization despite decent binary classification performance.
LLM agents are increasingly applied to vulnerability analysis, but existing benchmarks have not kept pace. They typically rely on small non-compilable snippets, focus on binary classification (vulnerable or not), and do not account for the risk that publicly-released datasets are part of model training corpora. We introduce RustMizan, a benchmarking framework for Rust vulnerability analysis that addresses these gaps. RustMizan contains compilable code variants at the crate, file, and function levels, with annotations for binary vulnerability detection, CWE classification, and function- and line-level localization. A paired mutation framework produces semantics-preserving code mutants for contamination testing and robustness probing. Across four frontier models in an agentic setup with command-line access, binary classification sits in the 56-65% range, but line localization F1 stays near 20%, and adversarial cues drop line F1 by about 27%.
Microsoft Research