Search papers, labs, and topics across Lattice.
The paper introduces HWE-Bench, a new benchmark to evaluate LLMs on board-level circuit design, a task requiring understanding of physics and IC datasheets. The benchmark comprises 300 design tasks and a knowledge base of 2,914 IC datasheets, evaluating LLMs on schematic generation from functional requirements and datasheets. Results show that while LLMs demonstrate usability and documentation understanding, they lack physical intuition, with the best model achieving only an 8.15% pass rate.
LLMs can read datasheets, but still can't design circuits, failing at basic physical intuition despite showing promise in documentation understanding.
Large Language Models (LLMs) have demonstrated significant potential in various engineering tasks, including software development, digital logic generation, and companion document maintenance. However, their ability to perform board-level circuit design is understudied, as this task requires a synergized understanding of real-world physics and Integrated Circuit (IC) datasheets, the latter comprising detailed specifications for individual components. To address this challenge, we propose \hweb, an evaluation framework that benchmarks the ability of LLMs to perform such designs. It consists of 300 board-level design tasks pulled from open-source and crowdsourcing platforms such as GitHub and OSHWLab, covering 8 application domains, and is complemented with a knowledge base of 2,914 real IC datasheets. For each task, the LLMs are tasked with generating a schematic from scratch, using the provided circuit functional requirements and a set of component datasheets as input. The resulting schematic will be checked against a static electrical rules, and then passed to a circuit simulator to verify its dynamic behavior. Our evaluation show that although current models achieve initial engineering usability and documentation understanding, they lack physical intuition, as the top-performing model achieved an overall pass rate of 8.15\%. We envision that advancements on \hweb\ will pave the way for the development of practical Electronic Design Automation (EDA) agents, revolutionizing the field of board-level design.
