Search papers, labs, and topics across Lattice.
This paper introduces LogicHunter, a fuzzing framework designed to enhance the testing of Large Language Model (LLM) agent frameworks by addressing the challenges of oracle ambiguity and input generation. By employing specification-driven generation and an innovative Agentic Oracle, LogicHunter systematically produces valid yet semantically extreme inputs, effectively exposing silent failures in the tested frameworks. The evaluation reveals that LogicHunter identified 40 previously unknown bugs, with a precision of 91.17%, significantly outperforming existing testing methods that reported no bugs.
LogicHunter uncovers 40 hidden bugs in LLM agent frameworks that traditional testing methods missed, achieving a groundbreaking 91.17% precision in bug detection.
Large Language Model (LLM) agent frameworks such as LangChain, LlamaIndex, and CrewAI have become critical infrastructure powering production AI systems, yet they remain severely under-tested due to fundamental challenges in automated testing. Unlike traditional software, where crashes serve as reliable oracles, defects in these pure Python frameworks manifest as ordinary exceptions or silent semantic failures, creating profound oracle ambiguity. This problem is exacerbated by strict type governance through Pydantic schemas and complex protocol requirements that cause existing fuzzers to generate overwhelming invalid inputs, while traditional test generators produce only trivial cases with weak regression assertions. We present LogicHunter, a fuzzing framework that addresses both the generation and oracle challenges through active specification-aware testing. LogicHunter employs specification-driven generation that systematically fuses formal type constraints with authentic usage patterns from real-world repositories, synthesizing inputs that are valid by construction yet semantically extreme, equipped with behavioral probes to expose silent failures. To resolve oracle ambiguity, we introduce the Agentic Oracle, which transcends passive classification by actively retrieving documentation, navigating source code, and inspecting runtime states through a ReAct-based architecture with Dual-Layer State Management and Dual-Stream Memory. Evaluated on three widely deployed frameworks, LogicHunter discovered 40 previously unknown bugs with 30 confirmed and 26 fixed by developers, while state-of-the-art baselines reported no bugs as final findings. The Agentic Oracle achieves 91.17% precision, surpassing the best passive approach at 29.27% by 61 percentage points.
