Search papers, labs, and topics across Lattice.
This paper introduces GateTruth, a mutation-testing engine designed to audit the rigor of RTL design benchmarks by injecting semantic mutants into reference designs to evaluate the effectiveness of testbenches. The authors validate their methodology using a dual-track reference suite, revealing that while 46 out of 60 testbenches effectively detect over 95% of mutants, a significant number fail to do so, highlighting the pitfalls of relying solely on testbench performance. Furthermore, the analysis of the widely-used RTLLM v2.0 benchmark shows that 72% of its designs do not meet the rigorous standards established by the authors, underscoring the need for improved benchmark validation practices in RTL design.
Mutation testing reveals that a staggering 72% of widely-used RTL benchmarks fail to meet basic rigor standards, challenging their reliability.
Benchmarks for evaluating large language models on register-transfer-level (RTL) hardware design have proliferated rapidly, yet none reports having applied mutation testing, an established hardware-verification technique for quantifying testbench quality, to ask whether its own testbenches are trustworthy. A testbench that never fails is not evidence of a correct design; it may simply never stimulate the logic that is actually broken. We introduce GateTruth, a mutation-testing engine and methodology for auditing RTL benchmark testbench rigor: inject a deterministic, seeded set of semantic mutants into a reference design and measure what fraction the testbench catches. We validate the methodology against our own 68-task, dual-track reference suite -- 60 specification-to-RTL generation tasks and 8 agentic-repair tasks, scored through a pinned, deterministic synthesis-to-timing flow with correctness enforced as a strict gate -- certifying that 46 of 60 Track A testbenches kill at least 95% of injected mutants under sequential, reproducible execution; we disclose why the other 14 do not, including a Goodhart effect on testbenches revised to pass this gate. We then point the same engine, unmodified, at RTLLM v2.0, a widely adopted external benchmark: of 46 auditable designs, 72% fall below the 95% floor our own suite is held to, and three score 0% outright. A comparable audit of NVIDIA's CVDP benchmark is structurally impossible: its public release withholds reference solutions, removing the golden RTL mutation testing requires. Auditing our own instrument also surfaced a second finding: an initially uniform 4096-token output cap silently truncated three of seven evaluated models, and re-running at 16,384 tokens moved one model from fifth place to first. We argue mutation-kill certification should become a standard reporting requirement for RTL-generation benchmarks generally.
