Search papers, labs, and topics across Lattice.
GAMMAF is introduced as an open-source benchmarking platform for evaluating graph-based anomaly detection methods in LLM multi-agent systems. It generates synthetic multi-agent interaction datasets using simulated debates across varied network topologies, capturing interactions as attributed graphs. Benchmarking involves evaluating defense models by isolating flagged adversarial nodes during live inference, demonstrating GAMMAF's utility, scalability, and efficiency using XG-Guard and BlindGuard baselines on MMLU-Pro and GSM8K.
LLM multi-agent systems can substantially reduce operational costs by using effective attack remediation to facilitate early consensus and cut off token generation by adversarial agents, as shown by GAMMAF.
The rapid integration of Large Language Models (LLMs) into Multi-Agent Systems (MAS) has significantly enhanced their collaborative problem-solving capabilities, but it has also expanded their attack surfaces, exposing them to vulnerabilities such as prompt infection and compromised inter-agent communication. While emerging graph-based anomaly detection methods show promise in protecting these networks, the field currently lacks a standardized, reproducible environment to train these models and evaluate their efficacy. To address this gap, we introduce Gammaf (Graph-based Anomaly Monitoring for LLM Multi-Agent systems Framework), an open-source benchmarking platform. Gammaf is not a novel defense mechanism itself, but rather a comprehensive evaluation architecture designed to generate synthetic multi-agent interaction datasets and benchmark the performance of existing and future defense models. The proposed framework operates through two interdependent pipelines: a Training Data Generation stage, which simulates debates across varied network topologies to capture interactions as robust attributed graphs, and a Defense System Benchmarking stage, which actively evaluates defense models by dynamically isolating flagged adversarial nodes during live inference rounds. Through rigorous evaluation using established defense baselines (XG-Guard and BlindGuard) across multiple knowledge tasks (such as MMLU-Pro and GSM8K), we demonstrate Gammaf's high utility, topological scalability, and execution efficiency. Furthermore, our experimental results reveal that equipping an LLM-MAS with effective attack remediation not only recovers system integrity but also substantially reduces overall operational costs by facilitating early consensus and cutting off the extensive token generation typical of adversarial agents.
