Search papers, labs, and topics across Lattice.
This study investigates the impact of large language models (LLMs) on the validity of hardware security competition benchmarks, revealing that LLMs can achieve an 83% detection rate through superficial syntactic comparisons rather than genuine security reasoning. To counter this threat, the authors introduce a novel obfuscation framework that applies human-readable transformations and controlled diff-noise, effectively preserving functionality while obscuring the underlying code. The framework significantly reduces LLM-based detection accuracy by 50% with minimal obfuscation and by 78.6% when fully applied, thereby restoring the reliability of these benchmarks for evaluating vulnerability detection methods.
LLMs can exploit syntactic patterns to falsely inflate detection rates in hardware security benchmarks, but a new obfuscation framework can slash their effectiveness by up to 78.6%.
Hardware security competitions such as HackTheSilicon serve as benchmarking platforms for evaluating vulnerability detection methods and for training humans and AI. However, our study reveals that LLMs threaten their validity. Instead of genuine security reasoning, detectors exploit a diff-style syntactic comparison, achieving an 83% detection rate, undermining fair evaluation. To mitigate this, we propose the first LLM-oriented, semantics-preserving obfuscation framework for these benchmarks. Unlike IP-protection approaches, it applies human-readable transformations and controlled diff-noise while preserving functionality. On HackTheSilicon, the framework reduces LLM-based detection accuracy by 50% with only 10% obfuscation and by 78.6% under complete obfuscation, restoring benchmark reliability.
