Search papers, labs, and topics across Lattice.
This paper investigates the challenges of zero-shot transfer in Vision-Language-Action (VLA) models for autonomous driving, particularly focusing on the limitations of training on individual datasets and the inefficacy of simply adding more datasets. The authors introduce BEV-Forcing, an auxiliary objective that integrates ground-plane object-layout information from a Bird's-Eye-View model into the VLA backbone, enhancing both in-distribution and out-of-distribution performance across multiple camera rigs. Key findings indicate that while BEV-Forcing improves performance with a limited number of embodiments, its advantages diminish as training diversity increases, highlighting the need for careful consideration of data scaling in model training.
BEV-Forcing reveals that integrating spatial information can significantly enhance VLA performance, but scaling up training diversity may nullify these gains.
Vision-Language-Action models (VLAs) have shown strong potential in autonomous driving by leveraging multimodal pretraining for instruction following, visual reasoning, and scene-level generalization. In robotic manipulation, scaling VLA fine-tuning across multiple robot setups--especially when unifying representations across embodiments--has been shown to improve in-dataset performance and cross-embodiment generalization; in autonomous driving, however, VLAs remain largely trained on individual datasets and are rarely evaluated for zero-shot transfer to unseen datasets and camera rigs; furthermore naively adding more datasets to the training data does not necessarily lead to better performance within seen embodiments. To address these problems, we study multi-dataset training for the driving task and BEV-Forcing, an auxiliary objective that transfers ground-plane object-layout information from a specialized Bird's-Eye-View model into the VLA backbone. By encouraging the model to represent object position through a shared BEV spatial interface, we show that an auxiliary task such as BEV-Forcing can improve both in-distribution and out-of-distribution performance when training on a small number of camera rigs. As the number of training embodiments increases, however, the benefits of the auxiliary task are reduced; we present this as evidence that new techniques in the literature may see their benefits diminish when simply scaling up training diversity, which motivates presenting results taking into account data scaling.
