Search papers, labs, and topics across Lattice.
This paper tackles the problem of weakly aligned language annotations in off-road autonomous driving datasets by proposing a language refinement framework that restructures annotations into action-aligned pairs for Vision-Language Models (VLMs). They introduce a preference optimization strategy using geometry-aware hard negatives to encourage terrain-aware planning. Experiments on the ORAD-3D benchmark show improvements in trajectory error, traversability compliance, and elevation consistency, demonstrating the effectiveness of their approach.
Fine-tuning VLMs with action-aligned language supervision and terrain-aware preference optimization unlocks more robust off-road autonomous driving, outperforming prior approaches on key traversability metrics.
While Vision-Language Models (VLMs) enable high-level semantic reasoning for end-to-end autonomous driving, particularly in unstructured environments, existing off-road datasets suffer from language annotations that are weakly aligned with vehicle actions and terrain geometry. To address this misalignment, we propose a language refinement framework that restructures annotations into action-aligned pairs, enabling a VLM to generate refined scene descriptions and 3D future trajectories directly from a single image. To further encourage terrain-aware planning, we introduce a preference optimization strategy that constructs geometry-aware hard negatives and explicitly penalizes trajectories inconsistent with local elevation profiles. Furthermore, we propose off-road-specific metrics to quantify traversability compliance and elevation consistency, addressing the limitations of conventional on-road evaluation. Experiments on the ORAD-3D benchmark demonstrate that our approach reduces average trajectory error from 1.01m to 0.97m, improves traversability compliance from 0.621 to 0.644, and decreases elevation inconsistency from 0.428 to 0.322, highlighting the efficacy of action-aligned supervision and terrain-aware optimization for robust off-road driving.
