Search papers, labs, and topics across Lattice.
This study investigates the integration of large language models (LLMs) with non-language agents in collaborative tasks, specifically through the introduction of a new method called latent state internalization. By utilizing \textsc{LLAMIA-Bench}, a benchmark suite for collaborative chess tasks, the authors demonstrate that traditional verbalization methods create a performance bottleneck, with a widening gap in effectiveness as LLMs scale. The results show that a single 14B parameter model, \textsc{LLAMIA}, trained with latent state internalization, outperforms both task specialists and advanced models like GPT-5.1 across all tasks, highlighting the advantages of direct representation integration.
Verbalization creates a performance bottleneck in LLM collaboration with non-language agents, but latent state internalization can bridge this gap, enabling superior task performance.
LLMs are increasingly deployed as orchestrators that coordinate specialized subagents to solve complex tasks through natural language. However, in many important domains like game playing and robotics, the strongest available agents are not language models. Integrating non-language agents with LLMs would require \emph{verbalization}: compressing their rich continuous representations into sparse textual summaries at each interaction step. To study whether verbalization constitutes a bottleneck, we introduce \textsc{LLAMIA-Bench}, a suite of six diverse collaborative chess tasks spanning three facets: behavioral imitation, state assessment, and natural-language explanation. Each task instantiates a well-established chess problem that neither the LLM nor the chess engine can solve alone. To solve LLM collaboration with non-language agents, we introduce \emph{latent state internalization}, which projects the subagent's continuous representations directly into the LLM's token stream as learned state tokens, with dynamic re-encoding as actions advance the environment state. Comparing internalization to verbalized integration, our experiments reveal a consistent \emph{verbalization debt}: the performance gap widens throughout training and persists as the LLM scales from 4B to 14B parameters. A single 14B model, \textsc{LLAMIA}, trained with latent state internalization, matches or exceeds task specialists and frontier models including GPT-5.1 with tool access across all benchmark tasks, and generalizes out-of-distribution where task-specific finetunes collapse
