Search papers, labs, and topics across Lattice.
This paper introduces a cost-aware selective inference framework for multimodal driver monitoring that integrates visual observations with physiological signals to enhance safety in automated vehicles. By employing a lightweight RGB-physiological model, the system achieves significant reductions in unsafe false negatives, dropping from 17.37% to approximately 5%, while maintaining low inference latency of 3.08ms. Additionally, the incorporation of a driver-state world modeling module provides predictive insights, although challenges in operating-point calibration remain a concern.
Cost-aware selective inference slashes unsafe false negatives in driver monitoring from 17.37% to around 5%, revolutionizing safety in automated vehicles.
Continuous driver monitoring in automated vehicles requires low-latency inference while avoiding unsafe decisions under uncertain driver states. Large vision-language models provide broad multimodal priors, but their latency and limited reliability in this setting make them unsuitable as always-on in-cabin monitors. We propose a cost-aware selective inference framework for deployable multimodal driver monitoring. The core system is a lightweight RGB-physiological student that combines in-cabin visual observations with window-level HR/EDA signals, and a learned gate that decides when to accept the fast prediction or abstain for safety intervention. Additional controls show that the learned scores contain sample-level information beyond scenario priors, while exact physiological synchronization remains a limitation. To incorporate predictive evidence, we further study a compact driver-state world modeling module that rolls out latent driver-state features and estimates future fast-model errors and counterfactual system-level action costs. On scenario-induced driver-demand recognition, the RGB-physiological student improves over RGB-only and physiology-only baselines, reaching 0.7440 Macro-F1 and 0.9099 balanced accuracy with 11.39M parameters and 3.08ms inference latency. Cost-aware selective inference reduces unsafe false negatives from 17.37% under always-fast inference to approximately 5% across seeds, while maintaining deployment-level latency. While driver-state world modeling offers valuable predictive signals, worst-group evaluations highlight persistent operating-point calibration drift. Ultimately, reliable edge driver monitoring requires advancing not only perception backbones, but also risk-aware selective control and group-robust calibration.
