Search papers, labs, and topics across Lattice.
This paper introduces MIRTH, a novel framework that enhances vision-language-action (VLA) models by integrating dual-scale temporal memory hubs, latent reasoning tokens optimized for mutual information, and a parallel action decoding scheme. These innovations address critical limitations in existing VLA architectures, such as temporal myopia and inefficient inference, allowing for improved alignment between multimodal context and action trajectories. Evaluations on both simulation and real-world platforms show that MIRTH not only achieves state-of-the-art performance but also demonstrates advanced error recovery capabilities, marking a significant advancement in robotic control.
MIRTH transforms VLA models by enabling them to effectively leverage historical dynamics and optimize action trajectories, leading to unprecedented error recovery in robotic control.
VLA models have emerged as a powerful paradigm for transferring semantic knowledge from web-scale data to physical robotic control. However, current single-frame architectures suffer from intrinsic limitations: temporal myopia that discards historical dynamics, reasoning gaps between high-level instructions and low-level motor commands, and inference inefficiency due to autoregressive scalar decoding. In this work, we propose MIRTH, a unified framework designed to address these challenges. MIRTH augments a pretrained VLA backbone with three key innovations: (1) dual-scale temporal memory hubs that compress long-term scene evolution and short-term motion trends into compact embeddings; (2) latent reasoning tokens optimized via a mutual-information objective carving out a semantic plan space to align multimodal context with action trajectories; and (3) a parallel action decoding scheme that replaces autoregressive generation with vector-wise prediction to maximize control throughput. Extensive evaluations on the LIBERO simulation benchmark and a real-world LeRobot platform demonstrate that MIRTH achieves state-of-the-art performance and exhibiting emergent error recovery capabilities. The codes and collected datasets are released at http://github.com/kiva12138/mirth.
