Search papers, labs, and topics across Lattice.
This paper introduces PixelPilot, a Vision-Language-Action model that decouples planning and lifting to enhance the scalability of autonomous driving systems. By reformulating trajectory prediction as sensor-agnostic 2D tasks, PixelPilot overcomes the limitations of traditional 2D-to-3D predictions, which are often hindered by camera parameter entanglement and convergence to trivial solutions. Extensive experiments show that PixelPilot achieves state-of-the-art performance in both open-loop and closed-loop settings, highlighting its superior scalability and visual reasoning capabilities across diverse driving datasets.
PixelPilot redefines trajectory prediction in autonomous driving by transforming it into scalable 2D tasks, leading to unprecedented generalization across heterogeneous datasets.
Vision-Language-Action Models (VLAs), which leverage the advanced reasoning capabilities of Vision-Language Models (VLMs), show promising generalization in complex autonomous driving scenarios. Existing VLAs typically predict and optimize 3D trajectories from 2D images. While intuitive, this 2D-to-3D prediction is inherently entangled with camera parameters, leading to limited data scalability across heterogeneous driving datasets. Moreover, directly optimizing in 3D space induces severe convergence to trivial solutions, where VLAs rely on ego-status rather than visual scene understanding. To address these issues, we propose PixelPilot, a novel VLA featuring a decoupled planning and lifting paradigm. In the planning phase, PixelPilot reformulates scene understanding and trajectory prediction as sensor-agnostic 2D-to-2D tasks in the image plane, thereby facilitating scalable training across diverse datasets. The planned 2D trajectories are then deterministically lifted to 3D only during inference, ensuring the full exploitation of visual cues and generalization across different vehicles. To realize this paradigm, we propose a knowledge-instilled policy learning strategy that applies dense, intermediate rewards via Group Relative Policy Optimization (GRPO) to enforce a rigorous causal chain from visual perception to spatial planning. Extensive experiments demonstrate that PixelPilot achieves state-of-the-art performance in both open-loop and closed-loop settings, validating its superior scalability and visual reasoning capabilities.
