Search papers, labs, and topics across Lattice.
The paper introduces CircuitReason-1k, a benchmark consisting of 1,000 authentic textbook problems designed to evaluate long-horizon visual-to-symbolic reasoning in electrical circuits. It emphasizes the complexity of circuit analysis, requiring models to ground symbols, recover topology, and formulate equations while adhering to physical conventions. Evaluation results show that while the best-performing model achieves 84.8% accuracy, significant performance drops occur on more complex problems, highlighting persistent challenges in topology binding and output propagation.
Despite achieving 84.8% accuracy, multimodal models struggle with long-horizon reasoning in electrical circuits, revealing critical gaps in their understanding of physical conventions.
Electrical circuit analysis requires more than recognizing components in an image. A solver must ground symbols and labels, recover latent topology, select a physical model, formulate coupled equations, propagate intermediate quantities, and preserve units, signs, directions, and phase conventions. We introduce \benchmark, a benchmark of 1,000 authentic textbook problems for evaluating this complete long-horizon visual-to-symbolic reasoning process. Each problem pairs one or more circuit diagrams with a self-contained question, a typed or semantically specified answer, and a reference worked solution. An evidence-first construction pipeline aligns questions, figures, and solutions, while a reasoning-oriented taxonomy organizes problems by circuit type and dependency depth. Evaluation combines conservative typed scoring with identity-blinded multi-model semantic consensus, retaining every problem in the denominator. Across three commercial chatbot systems and six open-source multimodal large language models, the highest-scoring system reaches 84.8\% accuracy. However, performance consistently deteriorates on long-horizon problems, and qualitative analysis exposes persistent failures in topology-to-target binding, physical conventions, and late-stage output propagation. \benchmark{} provides a focused testbed for measuring whether multimodal models can transform technical visual evidence into sustained, physically valid symbolic reasoning. Code are available at GitHub - CircuitReason/CircuitReason1K.
