Search papers, labs, and topics across Lattice.
OctoSense introduces a novel self-supervised learning framework for multimodal robot perception, leveraging a diverse sensor suite that includes stereo RGB, event cameras, LiDAR, and thermal imaging. The proposed method employs a late-fusion masked autoencoder that utilizes modality-specific tokenizers to effectively handle the varied spatiotemporal characteristics of the data, achieving superior performance on key robotics tasks compared to traditional image-only models. Key results demonstrate that OctoSense maintains robust prediction capabilities even in challenging conditions, such as nighttime or degraded sensor data, with impressive processing speeds on advanced hardware.
OctoSense outperforms conventional image-only models in multimodal robot perception, achieving robust performance even under degraded sensory conditions.
We present OctoSense, an open-source sensor platform with stereo RGB and event cameras, LiDAR, a thermal camera, an inertial measurement unit, RTK-corrected global positioning system, and proprioception (CAN bus data from a car, and joint angles for a quadruped robot). The eponymous OctoSense dataset contains 59 hours of time-synchronized driving data across different types of environments at different times of the day, including situations with highly degraded sensors. We demonstrate multi-modal self-supervised learning using such real-world robotics data, where sensors have different representations, frequencies, latencies and noise. Our approach, a "late-fusion" masked autoencoder, (i) uses modality-specific tokenizers to account for different spatiotemporal characteristics of these sensors, and (ii) caches modality-specific tokens at inference time to process new measurements as they come. This architecture (i) is fast (6.68 ms and 112 ms on NVIDIA 5090 and Orin NX respectively, to compute the representation), (ii) performs better than existing image-only foundation models on tasks such as estimation of optical flow, depth, semantic segmentation, and ego-motion (translation, rotation, and steering angle), and (iii) predicts robustly at nighttime or in situations where sensory data is degraded. See our project page for links to the dataset, code, and supplementary videos: https://abisulco.com/octosense/.
