Search papers, labs, and topics across Lattice.
The paper introduces GameDevBench, a new benchmark for evaluating multimodal agents on game development tasks, which require navigating complex codebases and manipulating multimodal assets. The benchmark comprises 132 tasks derived from web and video tutorials, demanding significant multimodal understanding and involving more extensive code modifications than previous software development benchmarks. Experiments reveal that current agents struggle with these tasks, and the benchmark highlights the correlation between task difficulty and multimodal complexity, while also demonstrating the effectiveness of simple image and video-based feedback mechanisms in improving agent performance.
Multimodal agents still struggle with game development, solving only ~50% of tasks in a new benchmark, GameDevBench, highlighting the need for better multimodal reasoning in complex software environments.
Despite rapid progress on coding agents, progress on their multimodal counterparts has lagged behind. A key challenge is the scarcity of evaluation testbeds that combine the complexity of software development with the need for deep multimodal understanding. Game development provides such a testbed as agents must navigate large, dense codebases while manipulating intrinsically multimodal assets such as shaders, sprites, and animations within a visual game scene. We present GameDevBench, the first benchmark for evaluating agents on game development tasks. GameDevBench consists of 132 tasks derived from web and video tutorials. Tasks require significant multimodal understanding and are complex -- the average solution requires over three times the amount of lines of code and file changes compared to prior software development benchmarks. Agents still struggle with game development, with the best agent solving only 54.5% of tasks. We find a strong correlation between perceived task difficulty and multimodal complexity, with success rates dropping from 46.9% on gameplay-oriented tasks to 31.6% on 2D graphics tasks. To improve multimodal capability, we introduce two simple image and video-based feedback mechanisms for agents. Despite their simplicity, these methods consistently improve performance, with the largest change being an increase in Claude Sonnet 4.5's performance from 33.3% to 47.7%. We release GameDevBench publicly to support further research into agentic game development.
CMU ML