Search papers, labs, and topics across Lattice.
This paper introduces StreamPI, a novel streaming multimodal temporal modeling framework that enhances Vision-Language-Action (VLA) models by enabling temporal reasoning without adding parameters. By employing instruction-anchored temporal modeling and a random-interval streaming training strategy, StreamPI effectively integrates past observations and maintains spatial perception during robot manipulation tasks. Experimental results show that StreamPI significantly outperforms the existing pi0.5 model across various real-robot tasks and simulation benchmarks, demonstrating its robustness and efficiency in asynchronous deployment scenarios.
StreamPI transforms VLA models by enabling them to retain temporal context and enhance spatial perception, outperforming traditional single-frame approaches.
Vision-Language-Action (VLA) models have demonstrated effectiveness in robot manipulation, yet state-of-the-art models such as pi0.5 operate under a single-frame paradigm, limiting their ability to retain past observations and develop precise spatial perception. In this paper, we propose StreamPI, a streaming multimodal temporal modeling framework that equips single-frame VLA with temporal reasoning capability without introducing any additional parameters. One core design is instruction-anchored temporal modeling. It treats each (visual observation, language instruction) pair as an atomic temporal unit: bidirectional attention within each pair enables cross-modal fusion, while causal attention across pairs preserves autoregressive streaming inference. This ensures the language instruction serves as a persistent semantic anchor throughout task execution. To bridge the gap between synchronous training and asynchronous real-robot deployment, we introduce a andom-interval streaming training strategy: a proper inter-frame interval (e.g., every 3 frames) enables faster and smoother action execution. Beyond this, randomizing the interval further improves robustness to frame-timing perturbations, supporting asynchronous deployment in practice. Furthermore, by leveraging the length extrapolation capability of the LLM backbone, StreamPI seamlessly inherits pretrained single-frame weights and supports flexible single-frame and multi-frame inference. Experiments on real-robot tasks spanning memory-dependent and precise perception scenarios, as well as the simulation benchmark LIBERO, demonstrate that StreamPI outperforms pi0.5 across diverse tasks.
