Search papers, labs, and topics across Lattice.
The paper introduces PearlVLA, a novel Vision-Language-Action framework that optimizes the trade-off between efficient action generation and explicit deliberation by refining action plans in latent space. By separating visual grounding and iterative planning into distinct branches, PearlVLA allows for low-latency execution while leveraging a lightweight frozen latent world model for enhanced planning accuracy. Empirical results on the LIBERO benchmark show that PearlVLA outperforms existing methods, achieving state-of-the-art performance in action generation tasks.
PearlVLA achieves state-of-the-art performance in action generation by refining plans in latent space, enabling low-latency execution without sacrificing deliberation quality.
Current Vision-Language-Action (VLA) models face a trade-off between efficient action generation and explicit deliberation. Directly decoding actions from vision-language backbone representations enables low-latency control, whereas explicit reasoning through textual chains, pixel-level subgoals, or action search can improve planning but incurs substantial latency and computational cost. We propose PearlVLA, a VLA framework that moves deliberation into the latent space of a vision-language model (VLM). PearlVLA separates VLM meta-query representations into a fixed visual grounding branch and an iterative latent plan branch. At each refinement round, a plan-conditioned world query probes a lightweight frozen latent world model for an action-free future observation latent, which is fed back to guide plan refinement. A future-guided RefineNet then applies scheduled residual updates to progressively refine a coarse semantic draft into a fine-grained latent action plan. The refined plan after K rounds is then decoded in parallel into an action chunk for low-latency execution. We further introduce Causal Refinement-Grouped Process-Reward RL to optimize the latent refinement process with rewards from longer-horizon imagined futures induced by latent plan edits. Empirical evaluations on the LIBERO benchmark demonstrate that PearlVLA achieves state-of-the-art performance among existing methods.
Tsinghua AI