Search papers, labs, and topics across Lattice.
The paper introduces SO3UFormer, a novel spherical transformer architecture designed to improve the rotation robustness of panoramic semantic segmentation models by learning intrinsic spherical features. It achieves this by decoupling feature representations from the gravity vector, employing quadrature-consistent spherical attention, and using a gauge-aware relative positional mechanism. Experiments on a new dataset variant, Pose35, and under full SO(3) rotations demonstrate that SO3UFormer significantly outperforms existing state-of-the-art methods in terms of mIoU, showcasing its stability under extreme rotations.
Standard panoramic segmentation models crumble when faced with real-world camera rotations, but SO3UFormer maintains high accuracy even under arbitrary 3D reorientations by learning rotation-invariant spherical features.
Panoramic semantic segmentation models are typically trained under a strict gravity-aligned assumption. However, real-world captures often deviate from this canonical orientation due to unconstrained camera motions, such as the rotational jitter of handheld devices or the dynamic attitude shifts of aerial platforms. This discrepancy causes standard spherical Transformers to overfit global latitude cues, leading to performance collapse under 3D reorientations. To address this, we introduce SO3UFormer, a rotation-robust architecture designed to learn intrinsic spherical features that are less sensitive to the underlying coordinate frame. Our approach rests on three geometric pillars: (1) an intrinsic feature formulation that decouples the representation from the gravity vector by removing absolute latitude encoding; (2) quadrature-consistent spherical attention that accounts for non-uniform sampling densities; and (3) a gauge-aware relative positional mechanism that encodes local angular geometry using tangent-plane projected angles and discrete gauge pooling, avoiding reliance on global axes. We further use index-based spherical resampling together with a logit-level SO(3)-consistency regularizer during training. To rigorously benchmark robustness, we introduce Pose35, a dataset variant of Stanford2D3D perturbed by random rotations within $\pm 35^\circ$. Under the extreme test of arbitrary full SO(3) rotations, existing SOTAs fail catastrophically: the baseline SphereUFormer drops from 67.53 mIoU to 25.26 mIoU. In contrast, SO3UFormer demonstrates remarkable stability, achieving 72.03 mIoU on Pose35 and retaining 70.67 mIoU under full SO(3) rotations.
