Search papers, labs, and topics across Lattice.
This study systematically investigates implementation bugs in Multi-Modal Agents (M-agents), revealing 158 unique bugs from 1,268 issue reports across 34 diverse agents. By developing a comprehensive taxonomy that categorizes these bugs based on symptoms and root causes, the authors highlight the specific challenges M-agents face in dynamic environments. The introduction of MATester, an automatic bug identifier, demonstrates its effectiveness by covering 61.4% of known issues and uncovering 31 new bugs in additional agents, underscoring the practical implications for enhancing M-agent reliability.
M-agents harbor 158 unique implementation bugs that could lead to critical failures in real-world applications, and a new tool can identify over 60% of these issues automatically.
Multi-Modal Agents (M-agents), empowered by Large Language Models (LLMs), excel in various complex, open-world scenarios such as autonomous driving and robotics. However, their unique requirements to interact with dynamic and diverse multi-modal environments introduce novel implementation challenges beyond those faced by traditional agents. Outdated perception, untrustworthy planning and inapplicable execution could cause traffic accident and financial loss. Despite growing study on agent issues, there has not been a systematic study focusing on M-agent-specific implementation bugs. To address this gap, we conducted the first systematic study of implementation bugs in M-agents. We collected 34 representative M-agents from diverse sources and, through meticulous filtering,identified 158 M-agent-specific bugs from 1,268 issue reports. Using a top-down strategy, we developed a comprehensive taxonomy that classifies bugs by global symptoms, functionality component-level symptoms, and root causes. We then implemented MATester, an automatic proof-of-concept bug identifier by analyzing runtime inter-component outputs. When applied to 12 extra M-agents, MATester successfully covered 61.4% of known open issues and discovered 31 additional bugs, demonstrating the practical usefulness of our study. Our work provides a comprehensive reference and guideline for classification, prevention and fix of M-agent bugs.
