Search papers, labs, and topics across Lattice.
The paper introduces CREW-Wildfire, a new open-source benchmark for evaluating LLM-based multi-agent systems in complex, dynamic, and partially observable wildfire response scenarios. The benchmark, built on the CREW simulation platform, features procedurally generated environments with large maps, heterogeneous agents, and long-horizon planning objectives. Experiments using state-of-the-art LLM-based multi-agent frameworks on CREW-Wildfire reveal significant performance gaps in coordination, communication, spatial reasoning, and long-horizon planning, highlighting challenges in scaling agentic AI.
Current LLM-based multi-agent systems struggle with large-scale coordination and long-horizon planning under uncertainty, as revealed by the new CREW-Wildfire benchmark.
Despite rapid progress in large language model (LLM)-based multi-agent systems, current benchmarks fall short in evaluating their scalability, robustness, and coordination capabilities in complex, dynamic, real-world tasks. Existing environments typically focus on small-scale, fully observable, or low-complexity domains, limiting their utility for developing and assessing next-generation multi-agent Agentic AI frameworks. We introduce CREW-Wildfire, an open-source benchmark designed to close this gap. Built atop the human-AI teaming CREW simulation platform, CREW-Wildfire offers procedurally generated wildfire response scenarios featuring large maps, heterogeneous agents, partial observability, stochastic dynamics, and long-horizon planning objectives. The environment supports both low-level control and high-level natural language interactions through modular Perception and Execution modules. We implement and evaluate several state-of-the-art LLM-based multi-agent Agentic AI frameworks, uncovering significant performance gaps that highlight the unsolved challenges in large-scale coordination, communication, spatial reasoning, and long-horizon planning under uncertainty. By providing more realistic complexity, scalable architecture, and behavioral evaluation metrics, CREW-Wildfire establishes a critical foundation for advancing research in scalable multi-agent Agentic intelligence. All code, environments, data, and baselines will be released to support future research in this emerging domain.
