Search papers, labs, and topics across Lattice.
This paper introduces RTSGameBench, a novel benchmark designed to evaluate the strategic reasoning capabilities of Vision-Language Models (VLMs) in real-time strategy (RTS) games. By leveraging the expansive gameplay of Beyond All Reason, RTSGameBench offers diverse evaluations and diagnostic assessments that target specific strategic competencies, addressing the shortcomings of existing benchmarks. The findings reveal that current state-of-the-art VLMs struggle significantly with coordination and multiagent tasks, particularly as the complexity of the scenarios increases.
VLMs falter in strategic reasoning, with performance dropping sharply in complex RTS scenarios that require tight coordination and long-horizon planning.
Modern Vision-Language Models (VLMs) often struggle with strategic reasoning, i.e., anticipating and influencing other agents' actions, under uncertainty in competitive and cooperative settings. Real-time strategy (RTS) games can be a natural testbed for diagnosing this limitation, as they demand coordination with allies, adaptation to opponents' strategy, and long-horizon planning under partial observability. However, existing RTS benchmarks offer limited evaluation scope, lack systematic competency diagnosis, and remain fixed in the pre-designed scenario coverage. To address these limitations, we present RTSGameBench, which is built on Beyond All Reason, a large-scale RTS game with an expanded battlefield that demands broader strategy diversity than the existing testbeds. The proposed benchmark provides evaluations through diverse gameplay across various matchup structures, diagnostic assessment via mini-games, each targeting an individual strategic competency, and extensible coverage via a self-evolving generation framework that converts free-form queries into new mini-games, improving over successive cycles. Additionally, for VLMs to operate in large-scale RTS games, we provide RTSGameAgent that manages units by an FSM with agentic memory. We empirically validate that multiple state-of-the-art VLMs do not perform well when matchups demand tighter coordination, multiagent coordination and when task scale increases.
