Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
To overcome partial-geometry ambiguities in category-level 6D pose estimation, the authors develop LEGAU, a framework that jointly optimizes NOCS correspondences, metric pose and size, and a canonical Semantic Gaussian Field. Unlike standard paradigms that isolate 3D shape reconstruction as a passive auxiliary loss, LEGAU actively feeds the category-conditioned Gaussian prior into a transformer-based fusion module to guide spatial reasoning across RGB-D observations and text embeddings. This unified pose-shape representation achieves state-of-the-art results in multi-category setups, delivering up to a 22% improvement on the SOPE benchmark alongside robust real-world transfer.
Treating 3D Gaussian reconstruction as an active structural prior rather than a passive auxiliary target drives a 22% performance leap in category-level 6D pose estimation.
Category-level 6D pose estimation from a single RGB-D observation is inherently under-constrained, since partial visible geometry must be interpreted together with a canonical object structure before a stable pose can be determined. We present LEGAU, a unified framework that jointly predicts NOCS correspondence, object pose and size, and a canonical Semantic Gaussian Field. Rather than treating reconstruction as a detached auxiliary task, LEGAU uses the Gaussian field as a category-conditioned structural prior that participates in multimodal feature fusion and provides global guidance for local pose reasoning. Conditioned on a categorical text embedding, LEGAU processes RGB-D observations through a transformer-based fusion module that integrates visual, geometric, and category-level cues, decoding the NOCS map, pose and size information and the Gaussian-based object representation. Extensive experiments on synthetic and real-world benchmarks show that this coupled pose-shape formulation achieves strong performance in a single-model multi-category setting, with up to 22\% on SOPE and competitive transfer to real-world data. These results highlight the benefit of jointly learning canonical correspondence, object shape, and pose alignment within a unified representation.
TU Munich