Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors introduce RTLBench, a multi-dimensional benchmark suite for evaluating LLM-generated RTL code across syntax, functionality, lint compliance, readability, and style consistency, using 160 cases from textbooks and open-source projects. They evaluate 24 state-of-the-art LLMs, revealing that while syntax and functionality are reasonably addressed, engineering quality aspects are often lacking. To improve LLM-generated RTL, they propose Log2BetterRTL, a log-driven feedback system that leverages EDA tool diagnostics to iteratively refine the code, demonstrating significant improvements in various quality metrics.
LLMs can generate RTL code, but they often fail on critical engineering quality metrics like lint compliance and readability, a gap RTLBench helps close.
The rapid advancement of large language models (LLMs) has enabled automated Register Transfer Level (RTL) code generation, accelerating chip design workflows. However, existing benchmarks focus mainly on syntax and functionality, overlooking critical engineering aspects such as lint compliance, readability, and coding style. To address this gap, we propose RTLBench, a benchmark suite of 160 copyright-free RTL cases sourced from textbooks and open-source projects. RTLBench features a multi-dimensional evaluation framework covering syntax, functionality, lint compliance, readability, and style consistency. To assess subjective code quality metrics, it also incorporates an LLM-as-a-judge mechanism. We evaluated 24 state-of-the-art LLMs using RTLBench, finding that while several models perform well in syntax and functionality, most fall short on engineering quality. To address this, we propose Log2BetterRTL, a log-driven feedback system that transforms EDA tool diagnostics into iterative improvement prompts. It improves syntax correctness by up to 18.13 %, boosts functional correctness by 14.38 %, reduces lint violations by up to 229, and raises clarity scores by 0.51. These results demonstrate RTLBench's effectiveness in evaluating and enhancing LLMgenerated RTL, bridging the gap between generative AI and industrial-grade hardware design. The suite and scripts are available at: https://fangzhigang32.github.io/RTLBench.