Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
To overcome visual degradation under extreme lighting and field-of-view loss in wrist-only robotic manipulation, the authors present ECHO, a latent world action model that encodes asynchronous event streams into compact spatio-temporal representations. The framework pairs a hindsight module—which tracks past event-conditioned gripper trajectories as addressable off-camera spatial context—with an outlook module that uses learnable foresight queries to anticipate future event windows. On RLBench benchmarks and physical robot setups, ECHO outperforms conventional RGB and RGB+event baselines by up to 20.6 percentage points under normal lighting and 14.6 points under extreme exposure drops, while notably surpassing third-person RGB camera setups.
Wrist-mounted event cameras can fully replace external camera rigs in robot manipulation, surviving extreme exposure failures by using past and predicted event streams to retain off-camera spatial context.
Learning-based manipulation policies relying on RGB cameras often suffer from degraded observations under extreme exposure. Event cameras mitigate this degradation by asynchronously detecting pixel-level intensity changes to offer a high dynamic range. However, their observations heavily depend on camera placement, as fixed cameras miss static scene content while wrist-mounted camera motion causes previously visited regions to leave the field of view. To address these spatial-temporal limitations, we present ECHO (Event-augmented Context with Hindsight and Outlook), a wrist-only latent world action model that encodes wrist events into compact motion representations to provide temporal and spatial context for policy reasoning. Specifically, ECHO utilizes a pretrained event encoder to explain visual-feature changes between frames. Its hindsight module preserves the gripper trajectory with past event stream as addressable off-camera context. Concurrently, the outlook module introduces learnable event foresight queries supervised to anticipate the event window for future actions, enabling the policy to predict upcoming scene changes. Evaluated on wrist-only RLBench tasks, ECHO outperforms RGB and RGB+event baselines by 20.6 and 12.0 percentage points under normal lighting, and by 14.6 and 11.3 points under severe exposure drops, respectively, while also surpassing RGB references using a third-person camera. Real-world experiments with a wrist-mounted event camera validate that ECHO outperforms RGB-only and RGB+event baselines across multiple tasks under both nominal and severely dark lighting. Project page is at https://echo-wam.github.io/.
CMU ML
Tsinghua AI