Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
mmHRI replaces privacy-invasive RGB cameras in human-robot interaction by using millimeter-wave (mmWave) radar to guide closed-loop vision-language-action (VLA) manipulation policies. To overcome radar sparsity and noise in cluttered environments, the framework couples a dual-stream network on raw radar tensors and point clouds with a memory-based state-space model for temporally coherent human action and 3D pose estimation. In occluded privacy-curtain environments, the system achieves 85.09% action-recognition accuracy and successfully executes closed-loop object delivery and retrieval across unseen subjects and scenes.
Assistive robots no longer require invasive visual cameras to track human gestures, successfully executing closed-loop manipulation through physical privacy curtains via mmWave radar and state-space temporal modeling.
Assistive robots increasingly operate in many human-centered environments and perform various human-robot interaction (HRI) tasks, such as object delivery. However, most existing HRI systems rely on RGB cameras that continuously observe humans to respond to non-verbal commands, such as hand gestures. This raises privacy concerns in privacy- critical environments, such as hospital wards or restaurants, where direct camera observation of humans is restricted. To develop privacy-preserving HRI, we leverage millimeter-wave (mmWave) radar, which can sense human motion through privacy barriers without identifiable imagery. We propose mmHRI, the first multi-modal robot manipulation framework that achieves mmWave radar-guided privacy-preserving HRI. mmHRI introduces two key designs to mitigate the sparsity and temporal inconsistency of radar data in cluttered robot manipulation environments. First, we propose a dual-stream architecture that jointly learns from unfiltered raw radar tensors and radar point clouds to estimate both human actions and 3D poses. To mitigate signal inconsistency, mmHRI further incorporates a memory-based state-space model (MSSM) that retains historical radar features to reduce abrupt changes in pose/action. These estimated human states are then converted into structured textual robot instructions, which control a vision-language-action (VLA) policy for closed-loop robot manipulation and human-aware reactions. Our evaluation covers human action recognition and closed-loop delivery and retrieval. In the privacy-preserving curtain setting, mmHRI achieves 85.09% action-recognition accuracy, outperforming existing radar-based alternatives. Robot trials further demonstrate successful delivery and retrieval under visual occlusion, with stable task performance across unseen subjects, clutter configurations, and environments.