Search papers, labs, and topics across Lattice.
This paper introduces a novel Latent Bridge that enhances real-time game agents by effectively coupling a slow reasoning model with a fast reactive model, optimizing the latency-quality tradeoff. By leveraging a learned continuous communication channel instead of traditional text-based interactions, the Latent Bridge significantly outperforms the standard Text Bridge across various gaming scenarios, achieving notable performance improvements in games like MsPacman and RoadRunner. The findings reveal that the Latent Bridge is particularly beneficial when the reasoning model already surpasses the reactive model, highlighting its potential for real-time applications in complex environments.
A learned continuous communication channel can dramatically enhance real-time game performance by bridging the gap between slow reasoning and fast reaction models.
A real-time agent for general computer use - with games as the most demanding case - must act within tens of milliseconds while still planning over seconds. These two regimes sit at opposite ends of the latency-quality tradeoff. A reasoning VLM (Qwen3-VL-8B-Thinking) deliberates effectively but requires ~1.5 s per response - far too slow for a 15 Hz control loop. In contrast, a reactive VLM (MiniCPM-o 4.5) acts in milliseconds but underperforms on planning-heavy tasks. We couple two frozen models of matched scale (9B reactive, 8B reasoning), leaving the communication channel as the sole trainable component. The standard coupling is a Text Bridge (T): the slow model writes a suffix the fast model reads. We introduce a learned continuous Latent Bridge (L) that projects the slow model's residuals into the fast model's input-embedding space in a LLaVA-style manner, avoiding any text round-trip; both are compared against Fast-Only (F). On 7 Atari games and a driving domain (MetaDrive), tuning the action decoder per channel on held-out seeds, the Latent Bridge matches or beats the Text Bridge in every domain: it significantly improves two games (MsPacman +57%, RoadRunner +28%) and is a safe drop-in elsewhere. Combining both channels interferes destructively (RoadRunner -96%), so only one should be used. The benefit is highly predictable: the bridge helps if and only if slow reasoning already beats fast reaction (T > F) - the Latent and Text gains over Fast-Only move together at r=0.93. MetaDrive is the controlled negative, where the Latent Bridge is demonstrably inert because the Text Bridge adds no value. We release replay recordings and reproducible pipelines.
UW