Search papers, labs, and topics across Lattice.
This paper introduces NetlistBench, a benchmark designed to evaluate the reliability of large language models (LLMs) in recognizing and manipulating SPICE netlists, which are crucial for circuit design. The study reveals that while simple local edits can achieve high accuracy (96%-100%), more complex tasks like device addition and equivalence judgment show significant performance drops, highlighting the challenges posed by structural complexity. The findings indicate that despite improvements from enabling reasoning in weaker models, structure-preservation failures persist, marking netlist reliability as a critical issue for LLM applications in circuit design automation.
LLMs struggle with complex SPICE netlist tasks, achieving only 41% accuracy on device addition despite near-perfect performance on simpler edits.
Large Language Models (LLMs) are increasingly used in circuit design workflows, yet their reliability on simulator-facing SPICE netlist recognition and manipulation remains poorly understood and is rarely separated from high-level design reasoning. Although netlists are textual, they encode structured circuit objects through topology and parameters. We present \textbf{NetlistBench}, a structure-verified benchmark for SPICE netlist recognition and manipulation. NetlistBench contains 2,342 cases across 24 task families, covering parameter and connectivity recognition and edits, hierarchical operations, equivalence judgment, and long-horizon compound editing. Model outputs are evaluated by a deterministic structure-aware oracle. Across six non-thinking LLMs, performance varies substantially with operation-level structural complexity. Simple local edits reach $96\%$--$100\%$ accuracy, while device addition drops to $41\%$--$83\%$ and equivalence judgment to $49\%$--$90\%$. Enabling reasoning substantially improves weaker models but does not eliminate structure-preservation failures, with performance still degrading sharply as the edit horizon increases. NetlistBench identifies netlist reliability as a distinct bottleneck for trustworthy LLM-based circuit design automation.
