Search papers, labs, and topics across Lattice.
The authors introduce BATON, a large-scale multimodal dataset of naturalistic driving scenarios with automation transitions, capturing synchronized front-view video, in-cabin video, CAN bus signals, radar data, and GPS route context across 127 drivers and 136.6 hours of driving. They define and benchmark three tasks: driving action understanding, handover prediction, and takeover prediction, using sequence models, classical classifiers, and zero-shot VLMs. Results highlight the necessity of multimodal input for reliable transition prediction, revealing an asymmetry between handover and takeover events with implications for HMI design.
Predicting when drivers will engage or disengage driving automation requires more than just video鈥擟AN bus data and route context are key, and handover/takeover events exhibit distinct temporal dependencies.
Existing driving automation (DA) systems on production vehicles rely on human drivers to decide when to engage DA while requiring them to remain continuously attentive and ready to intervene. This design demands substantial situational judgment and imposes significant cognitive load, leading to steep learning curves, suboptimal user experience, and safety risks from both over-reliance and delayed takeover. Predicting when drivers hand over control to DA and when they take it back is therefore critical for designing proactive, context-aware HMI, yet existing datasets rarely capture the multimodal context, including road scene, driver state, vehicle dynamics, and route environment. To fill this gap, we introduce BATON, a large-scale naturalistic dataset capturing real-world DA usage across 127 drivers, and 136.6 hours of driving. The dataset synchronizes front-view video, in-cabin video, decoded CAN bus signals, radar-based lead-vehicle interaction, and GPS-derived route context, forming a closed-loop multimodal record around each control transition. We define three benchmark tasks: driving action understanding, handover prediction, and takeover prediction, and evaluate baselines spanning sequence models, classical classifiers, and zero-shot VLMs. Results show that visual input alone is insufficient for reliable transition prediction: front-view video captures road context but not driver state, while in-cabin video reflects driver readiness but not the external scene. Incorporating CAN and route-context signals substantially improves performance over video-only settings, indicating strong complementarity across modalities. We further find takeover events develop more gradually and benefit from longer prediction horizons, whereas handover events depend more on immediate contextual cues, revealing an asymmetry with direct implications for HMI design in assisted driving systems.
