Search papers, labs, and topics across Lattice.
The authors introduce GameWorld, a benchmark for evaluating multimodal large language model (MLLM) agents in video game environments, addressing the challenges of heterogeneous action interfaces and heuristic verification. GameWorld includes 34 diverse games and 170 tasks with state-verifiable metrics, enabling outcome-based evaluation of agents using either computer-use or semantic action parsing interfaces. Experiments across 18 model-interface pairs reveal that current agents are far from human-level performance, highlighting challenges in real-time interaction, context-memory sensitivity, and action validity.
Current multimodal agents are nowhere near human-level gaming, even when given precise semantic actions, revealing critical gaps in real-time interaction and context handling.
Towards an embodied generalist for real-world interaction, Multimodal Large Language Model (MLLM) agents still suffer from challenging latency, sparse feedback, and irreversible mistakes. Video games offer an ideal testbed with rich visual observations and closed-loop interaction, demanding fine-grained perception, long-horizon planning, and precise control. However, systematically evaluating these capabilities is currently hindered by heterogeneous action interfaces and heuristic verification. To this end, we introduce GameWorld, a benchmark designed for standardized and verifiable evaluation of MLLMs as generalist game agents in browser environments. Two game agent interfaces are studied: (i) computer-use agents that directly emit keyboard and mouse controls, and (ii) generalist multimodal agents that act in a semantic action space via deterministic Semantic Action Parsing. GameWorld contains 34 diverse games and 170 tasks, each paired with state-verifiable metrics for outcome-based evaluation. The results across 18 model-interface pairs suggest that even the best performing agent is far from achieving human capabilities on video games. Extensive experiments of repeated full-benchmark reruns demonstrate the robustness of the benchmark, while further studies on real-time interaction, context-memory sensitivity, and action validity expose more challenges ahead for game agents. Together, by offering a standardized, verifiable, and reproducible evaluation framework, GameWorld lays a robust foundation for advancing research on multimodal game agents and beyond. The project page is at https://gameworld-bench.github.io.
