Search papers, labs, and topics across Lattice.
This paper introduces $\mathrm{ACT}^3$, an Action-Centric Tri-Stream Transformer designed to improve Vision-Language-Action (VLA) models for robotic manipulation by effectively fusing semantic understanding from VLMs and physical dynamics priors from World Models. The model achieves this by enabling a dedicated action expert to access VLM and WM representations via layerwise attention within distinct streams, preserving independent forward propagation while allowing joint updates through control supervision. Experimental results on simulated and real-world robotic manipulation tasks demonstrate $\mathrm{ACT}^3$'s superior performance compared to existing methods.
A novel tri-stream transformer architecture unlocks synergistic control by enabling a dedicated action expert to selectively fuse semantic and dynamics information from independent VLM and World Model streams.
Vision-Language-Action (VLA) models have emerged as a prominent framework for complex robotic manipulation, building on the strong semantic understanding of pretrained Vision-Language Models (VLMs). However, such VLM backbones offer insufficient physical dynamics priors, which limits the generalization capabilities of robot policies. Recent efforts therefore integrate video-generation World Models (WMs) into robot policies through various strategies, using predictive dynamics to facilitate action generation. Despite these advances, harnessing semantic understanding and dynamics prediction as complementary guidance for action generation remains challenging. In this paper, we introduce $\mathrm{ACT}^3$, a simple yet effective Action-Centric Tri-Stream Transformer that fuses semantic and dynamics information into control actions while preserving the distinct roles of context streams. Specifically, $\mathrm{ACT}^3$ enables the dedicated action expert to access VLM and WM representations through layerwise attention, with each backbone attending only within its own stream. This straightforward interaction design maintains independent forward propagation in the context streams while allowing both backbones to be updated through control supervision. Experiments on both simulated and real-world robotic manipulation benchmarks show that the proposed $\mathrm{ACT}^3$ yields results superior to its counterparts.
