Search papers, labs, and topics across Lattice.
This paper introduces a streaming Gaussian encoder that enhances 4D panoptic occupancy tracking (4D-POT) by maintaining a persistent volumetric scene representation, addressing the limitations of traditional mask-based pipelines that suffer from temporal inconsistencies. By employing ego-motion compensation and a confidence-guided budget constraint, the method effectively propagates latent Gaussian queries while ensuring geometric coherence, particularly in the presence of occlusions and static elements. Extensive evaluations on the Occ3D-extended nuScenes and Waymo datasets demonstrate that this approach achieves state-of-the-art tracking consistency with minimal computational overhead, integrating seamlessly with existing frameworks.
Achieving state-of-the-art 4D panoptic occupancy tracking, this method maintains temporal coherence in scene representations without the computational burden of recomputing volumetric data at each timestep.
Camera-based 4D panoptic occupancy tracking (4D-POT) is a promising paradigm for holistic scene understanding from multi-view imagery, enabling joint reasoning about geometry, semantics, and object identities across time. Recent mask-based pipelines achieve strong performance by propagating instance queries across frames. However, their underlying volumetric representations are typically recomputed at each timestep, limiting geometric temporal consistency, particularly under occlusion and for static scene elements. To address this limitation, we propose a streaming Gaussian encoder that maintains a persistent volumetric scene representation for 4D-POT. Our method models the scene as a fixed-size set of latent Gaussian queries that are propagated via ego-motion compensation and refreshed under a confidence-guided budget constraint. Crucially, we shape Gaussian opacities through depth-based supervision to serve as proxy for visibility, enabling confidence to accumulate as a temporally aggregated measure of persistent scene support. Together with a warmup-based multi-frame training strategy, this yields representation-level temporal coherence beyond decoder-only tracking. Extensive experiments on Occ3D-extended nuScenes and Waymo establish a new state-of-the-art for camera-based 4D-POT, improving tracking consistency with negligible computational overhead while remaining fully compatible with existing mask-based pipelines. We provide code and models at https://sge.cs.uni-freiburg.de.
