Search papers, labs, and topics across Lattice.
This paper introduces Lagrange, an innovative framework for end-to-end autonomous driving that leverages Masked Latent Fields to achieve open-vocabulary reasoning while maintaining kinematic validity. By utilizing Vision-Language Models to create continuous semantic visual tokens and an intent-driven masked cross-attention module, Lagrange effectively balances representational efficiency and generalization capacity, addressing the limitations of both dense and sparse models. Extensive evaluations on standard and long-tail benchmarks reveal that Lagrange not only enhances robustness in complex driving scenarios but also ensures interpretable decision-making aligned with vehicle dynamics.
Lagrange achieves robust open-world driving by transforming decision-making into a Lagrangian action minimization problem, ensuring both interpretability and compliance with vehicle kinematics.
Scaling end-to-end autonomous driving to complex, open-world environments requires perceptual models that generalize to anomalous scenarios and planners that produce kinematically valid trajectories. Existing paradigms face a distinct dichotomy between representational efficiency and generalization capacity. Dense models (e.g., occupancy networks), while geometrically robust, incur critical computational bottlenecks and struggle with high-level semantic reasoning. Conversely, sparse, query-based planners are efficient but reliant on closed-set definitions, rendering them vulnerable to out-of-distribution (OOD) events. Although recent Vision-Language-Action (VLA) models offer open-vocabulary reasoning, their autoregressive, discrete token generation fundamentally conflicts with the continuous, high-frequency control requirements of vehicle dynamics. To address this, we propose Lagrange, an open-vocabulary, computationally sparse driving framework based on Masked Latent Fields (MLF). Rather than relying on dense volumetric reconstructions or closed-set query mechanisms, Lagrange exploits Vision-Language Models (VLMs) to encode class-agnostic object proposals into continuous semantic visual tokens. We introduce an intent-driven masked cross-attention module that temporally filters irrelevant entities, decoding the attended tokens into an implicit continuous energy field defined over spatial coordinates. By framing decision-making as a Lagrangian action minimization problem spanning this energy field, we enforce strict compliance with vehicle kinematics while executing collision avoidance. Extensive offline evaluations on both standard (nuScenes) and long-tail (CODA) benchmarks demonstrate that Lagrange establishes a promising framework for robust, interpretable, and kinematically feasible open-world autonomy.
