Search papers, labs, and topics across Lattice.
The paper introduces MAS-FIRE, a framework for fault injection and reliability evaluation of LLM-based multi-agent systems, addressing the challenge of semantic failures that propagate silently in these systems. MAS-FIRE defines a taxonomy of 15 fault types and injects them using prompt modification, response rewriting, and message routing manipulation. Experiments on three MAS architectures reveal fault-tolerant behaviors organized into four tiers, demonstrating that stronger foundation models do not guarantee robustness and that architectural topology significantly impacts fault tolerance.
LLM-based multi-agent systems can neutralize 40% more faults with iterative, closed-loop designs compared to linear workflows, revealing architectural topology as a key factor in robustness.
As LLM-based Multi-Agent Systems (MAS) are increasingly deployed for complex tasks, ensuring their reliability has become a pressing challenge. Since MAS coordinate through unstructured natural language rather than rigid protocols, they are prone to semantic failures (e.g., hallucinations, misinterpreted instructions, and reasoning drift) that propagate silently without raising runtime exceptions. Prevailing evaluation approaches, which measure only end-to-end task success, offer limited insight into how these failures arise or how effectively agents recover from them. To bridge this gap, we propose MAS-FIRE, a systematic framework for fault injection and reliability evaluation of MAS. We define a taxonomy of 15 fault types covering intra-agent cognitive errors and inter-agent coordination failures, and inject them via three non-invasive mechanisms: prompt modification, response rewriting, and message routing manipulation. Applying MAS-FIRE to three representative MAS architectures, we uncover a rich set of fault-tolerant behaviors that we organize into four tiers: mechanism, rule, prompt, and reasoning. This tiered view enables fine-grained diagnosis of where and why systems succeed or fail. Our findings reveal that stronger foundation models do not uniformly improve robustness. We further show that architectural topology plays an equally decisive role, with iterative, closed-loop designs neutralizing over 40% of faults that cause catastrophic collapse in linear workflows. MAS-FIRE provides the process-level observability and actionable guidance needed to systematically improve multi-agent systems.
