Search papers, labs, and topics across Lattice.
This paper introduces WaveformQA, a novel open-source benchmark designed to evaluate the temporal reasoning capabilities of large language models (LLMs) on digital waveform data, addressing a critical gap in existing benchmarks that primarily focus on HDL code generation. The benchmark consists of 360 questions across eight difficulty categories, targeting complex tasks such as multi-signal correlation and event ordering, and is grounded in real hardware behavior through programmatically generated waveforms. Evaluation results indicate that while LLMs perform adequately on simpler queries, their accuracy significantly declines on more complex temporal reasoning tasks, highlighting the limitations of current models in this domain and the advantages of using an event-time JSON representation over traditional VCD formats.
LLMs struggle with complex temporal reasoning, showing significant accuracy drops on intricate waveform queries, underscoring a critical gap in their capabilities.
Large Language Models (LLMs) have demonstrated strong capabilities in code generation and reasoning, yet their ability to perform temporal reasoning over digital waveform data remains largely unexplored. Although reasoning over digital waveforms is a critical bottleneck in design verification, existing benchmarks primarily evaluate hardware description language (HDL) code generation and use waveforms only as supplementary context. This paper presents WaveformQA, an open-source question-answering benchmark for evaluating LLM temporal reasoning over digital waveforms. The benchmark comprises 360 questions with programmatically generated ground truths across eight categories of varying difficulty, including questions targeting multi-signal correlation and event ordering. Waveforms are generated from open-source design implementations, ensuring reproducibility and grounding the benchmark in real hardware behavior. Evaluation of frontier LLMs reveals that while models achieve reasonable accuracy on simple queries, performance degrades due to context window limitations and reasoning difficulties on complex temporal and multi-step questions. In addition, we show that an event-time JSON representation of waveforms improves LLM reasoning accuracy versus the standardized value change dump (VCD) format. The open-source framework supports extending to new question categories and importing new waveform sources, enabling researchers to rapidly prototype temporal reasoning experiments.
