Search papers, labs, and topics across Lattice.
The paper introduces VLGA, a novel vision-language-action model that incorporates geometry as a fourth modality to enhance autonomous driving capabilities. By employing a per-pixel pointmap regression loss against LiDAR data, VLGA effectively reconstructs the dense 3D environment, allowing for more accurate action grounding in complex driving scenarios. Experimental results on the nuScenes and Bench2Drive datasets demonstrate VLGA's superiority, achieving state-of-the-art performance in both open-loop and closed-loop evaluations, with significant improvements in collision rates and driving scores compared to existing VLA methods.
VLGA sets a new benchmark in autonomous driving by integrating dense 3D geometry, achieving unprecedented accuracy and safety in complex environments.
Vision-language-action (VLA) models can describe scenes and reason about them in language, yet still struggle to ground their actions in the dense 3D world around them. Existing approaches either inject features from a frozen 3D foundation model without an objective that ensures the policy uses them, or constrain geometry with sparse box and map losses that provide no dense spatial signal. We introduce VLGA, the first vision-language-action model supervised to reconstruct the dense 3D world it drives through. VLGA introduces geometry as a fourth modality alongside vision, language, and action through a dedicated expert supervised by a per-pixel pointmap regression loss against LiDAR. Extensive experiments conducted on challenging nuScenes and Bench2Drive datasets for open-loop and closed-loop evaluations, respectively, show the superiority of VLGA over counterpart VLA methods. In particular, on open-loop nuScenes, VLGA sets a new state of the art among VLA methods without ego status, with the lowest L2 (0.50\,m average) and 3-second collision rate (0.18\%). On closed-loop Bench2Drive, VLGA attains the state-of-the-art driving score of 79.08, +0.71 over the strongest prior VLA, at comparable efficiency and comfort.
