Search papers, labs, and topics across Lattice.
This paper introduces WalkOCC, a novel hybrid Ray-marching monocular 3D occupancy perception framework tailored for mobile robots navigating crowded sidewalks. By integrating geometric grounding from LiDAR-RGB paired data with scalable learning from unpaired monocular images, WalkOCC achieves improved prediction accuracy and fine-grained segmentation of urban features without the need for extensive 3D occupancy annotations. Experimental results show that WalkOCC outperforms existing self-supervised image-based methods, demonstrating robust performance across various environmental conditions and robot embodiments.
WalkOCC achieves superior sidewalk occupancy prediction by leveraging unpaired monocular images, eliminating the need for costly 3D annotations.
Sidewalks in the real world are crowded, cluttered, and less structured than roads, making 3D occupancy prediction a key ingredient for the safe navigation of mobile robots such as delivery bots and electric wheelchairs. Existing occupancy learning pipelines are largely designed for on-road autonomous driving and often train on large-scale paired LiDAR-RGB datasets with dense 3D supervision and multiple camera inputs, which are costly to collect and do not adequately capture sidewalk-specific characteristics. We propose WalkOCC, a hybrid Ray-marching monocular 3D occupancy perception framework for robots operating on sidewalks. WalkOCC explicitly couples geometric grounding from LiDAR-RGB paired data with scalable learning from large-scale unpaired monocular images. It bootstraps pseudo occupancy supervision from paired sequences and jointly learns image-level representations on additional 2D-only data. It yields stable optimization and improved generalization without requiring costly 3D occupancy annotations. Extensive experiments demonstrate consistent gains in prediction accuracy, fine-grained segmentation of subtle urban structures such as curbs and gutters, and robustness to environmental and cross-embodiment shifts compared with self-supervised image-based baselines. To facilitate evaluation and benchmarking, we also introduce Sidewalk3D, a large-scale sidewalk perception dataset with LiDAR-camera paired sequences collected across multiple locations and time periods, along with 3D semantic occupancy annotations for evaluation. Code and data will be made available.
MIT CSAIL