Search papers, labs, and topics across Lattice.
This paper introduces GameCraft-Bench, a novel benchmark designed to evaluate the ability of coding agents to generate playable games end-to-end within a game engine, specifically targeting the Godot platform. The authors establish a framework that emphasizes Engine Grounding, Artifact Completeness, and Interactive Verification to assess the quality of generated games through player interactions and multimodal evaluations. Despite the advancements in coding agents, the results reveal that even the best-performing agent only achieves a score of 41.46%, indicating significant challenges in producing fully functional and engaging game experiences.
Coding agents struggle to create complete and engaging games, with top performers barely reaching 41.46% success in end-to-end game generation.
Game generation is an emerging application of coding agents, requiring models to transform natural-language specifications into playable interactive systems. Unlike traditional coding tasks, game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay. We formalize end-to-end game generation as the problem of producing a complete game artifact that realizes a specification through observable player-game interaction in a target environment. We argue that evaluating this setting requires three desiderata: Engine Grounding, Artifact Completeness, and Interactive Verification. We propose an interaction-grounded evaluation framework that assesses executable gameplay through replayed demonstrations and rubric-guided multimodal judging. We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families. Evaluations of frontier coding agents show that end-to-end game generation remains highly challenging: the strongest agent achieves only 41.46%, and most agents score below 40%. Further analysis reveals that while agents often implement recognizable mechanics, they struggle to deliver complete games with sufficient content, functional visual feedback, and coherent presentation. See https://tongxuluo.github.io/gamecraft-bench-website for demos, code, and data.
NUS