Search papers, labs, and topics across Lattice.
This paper introduces the Latent Memory Palace (LMP), a novel approach that organizes information in an autoregressive latent space to enhance reasoning for continuous control policies. By framing reasoning as variational inference, the authors derive a reinforcement learning technique that optimizes the variational lower bound, leading to a policy that demonstrates strong performance in both simulated and real-world environments. The framework also includes a variable-length action tokenizer, significantly boosting the efficacy of downstream autoregressive policies while providing interpretable and adaptive compute allocation during test time.
Reasoning for control can be transformed into an adaptive, iterative process that leverages a latent memory structure, yielding superior performance in complex tasks.
Human decision-making is highly flexible -- some actions are taken immediately; others require longer deliberation. Language models have exhibited a similar capacity for adaptive"reasoning."However, transferring this capability to continuous control policies has been challenging, as directly reasoning in language space may lack the granularity for spatial understanding and precise motions. In this work, we show that reasoning for control policies can emerge by organizing information in an autoregressive latent space reminiscent of a memory palace, where retrieval is iterative and adaptive. Our method, Latent Memory Palace (LMP), formulates reasoning as variational inference with an autoregressive latent distribution. We derive a latent-space reinforcement learning technique to tractably optimize its variational lower bound. The resulting policy, LMP-$\pi$, achieves strong empirical performance in simulation and real-world domains while exhibiting interpretable, adaptive allocation of test-time compute. We further show that the same framework yields a variable-length action tokenizer, LMP-$\texttt{tok}$, which significantly improves the performance of downstream autoregressive policies. Together, these results present a new perspective on latent reasoning for control through the lens of variational inference.
UW