Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To address the gap between reactive driving agents and complex passenger interactions, the authors introduce doPlan, a dataset of 5,154 human annotations over 50.9 hours of nuPlan driving data capturing deferred, multi-stage, and event-conditioned instructions across variable horizons up to 508 seconds. This persistent context is critical because passenger requests rarely map to immediate maneuvers, requiring autonomous planners to track unresolved goals across evolving driving environments. Evaluating four language-conditioned planners reveals a fundamental horizon mismatch: the median instructed maneuver occurs 24.6 seconds after the prompt, meaning only 9.8% fall within conventional 5-second prediction horizons.
Standard 5-second planning horizons miss over 90% of real-world passenger instructions, exposing a severe architectural failure mode in language-conditioned autonomous driving.
Autonomous vehicles interacting with passengers through natural language must reason beyond immediate commands. Passenger intent may span multiple stages of behavior, depend on future events, refer to surrounding agents or landmarks, and remain relevant as driving conditions evolve. Existing language-enabled driving datasets largely focus on short, localized interactions, leaving these longer-horizon forms of passenger intent comparatively underexplored. We introduce doPlan, to our knowledge the first publicly available, human-annotated real-world dataset designed to study passenger language as persistent task context. Built on nuPlan, doPlan contains 5,154 human-written passenger instructions spanning 169.1 hours of cumulative instruction-aligned context over 50.9 hours of unique driving, with annotation windows ranging from 30.0 to 508.8 s. The annotations capture immediate, deferred, event-conditioned, persistent, and multi-stage passenger intent. The dataset, annotation interface, and supporting resources are publicly available at https://github.com/Mi3-Lab/doPlan. We evaluate four language-conditioned driving models and find that sensitivity to passenger language does not reliably translate into behavior consistent with the requested direction. More broadly, among 2,161 examples with a matched future maneuver, the first associated maneuver occurs a median of 24.6 s after the evaluation point, and only 9.8% occur within the models' common 5 s prediction horizon. These findings highlight the need to connect persistent passenger intent with successive planning decisions. doPlan provides a setting for studying how unresolved goals can be retained, grounded in evolving scenes, and tracked across multiple stages, including how a planner determines when a future goal becomes relevant to the current plan.