Search papers, labs, and topics across Lattice.
This paper introduces LISA, an LLM-based invariant testing framework designed to identify functional bugs in software libraries by generating API sequences and program invariants through API n-gram feedback. Unlike traditional fuzzing methods that primarily detect crashes, LISA effectively uncovers functional bugs that do not lead to program failures, thereby addressing a significant gap in existing testing methodologies. The results demonstrate that LISA achieves higher bug-detection rates and competitive code coverage compared to both conventional fuzzing techniques and previous LLM-based testing approaches.
LISA uncovers functional bugs that traditional testing methods miss, achieving superior detection rates without relying on crashes.
Manually writing unit tests to uncover functional bugs in software libraries is not only time-consuming but also requires a deep understanding of the intended semantics of the APIs. Heuristic-based test generation methods suffer from low usability because they cannot reason about program semantics or interpret source code and documentation as humans do. Traditional fuzzing techniques like OSS-Fuzz often rely on crashes to detect bugs, but functional bugs do not always cause crashes. To overcome these limitations, we present LISA, a novel LLM-based invariant testing framework for software functional bugs. LISA iteratively generates API sequences and program invariants guided by API n-gram feedback, achieving higher bug-detection rates and competitive code coverage compared with both fuzzing and prior LLM-based test generation approaches, and reporting each finding as a high-confidence bug candidate for developer confirmation.
