Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
ExcavaTwin establishes a training-free, pure-vision semantic elevation mapping framework for autonomous heavy machinery using frozen vision foundation models. Addressing the failure of generic semantic models in unstructured outdoor environments, the approach applies geometric support constraints across multi-view RGB observations to eliminate hallucinated semantics and recover incomplete terrain profiles. In real-world dynamic excavation trials, the pipeline maintained robust spatio-temporal mapping with a 12.74 cm mean elevation error and a 1.4-second update interval.
Zero-shot vision foundation models can guide heavy machinery through dynamic, unstructured earthmoving environments down to 12.7 cm elevation accuracy without a single task-specific training label.
Autonomous excavation requires a spatial representation that jointly captures terrain geometry and task-relevant semantics. Existing excavation mapping is largely elevation-centric, while generic semantic models remain unstable in unstructured outdoor scenes. We present ExcavaTwin, a pure-vision geometry-guided semantic elevation mapping framework without excavation-specific training. Given multi-view RGB images, the framework: 1) reconstructs scene geometry and semantic observations using frozen vision models; 2) derives terrain and non-terrain geometric support; 3) performs geometry-constrained multi-view semantic fusion to suppress implausible predictions and recover incomplete observations; and 4) projects the fused state into a task-oriented semantic elevation map. Experiments on public datasets and real excavation scenes demonstrate reliable geometric and semantic perception. In real excavation, the system achieved an average update interval of approximately 1.4 s and a mean elevation error of 12.74cm in dynamically modified regions. Larger errors mainly occur during rapid terrain changes and transient visual disturbances caused by machine motion.