Search papers, labs, and topics across Lattice.
This paper introduces PLANET, an innovative multi-object tracker that transcends traditional image-plane limitations by incorporating 3D scene geometry into its tracking process. By lifting 2D tracking datasets into 3D and utilizing world-grounded queries, PLANET enhances object localization and association, addressing ambiguities inherent in monocular video data. The approach, bolstered by a dual-resolution temporal memory and an auxiliary 3D location prediction task, results in state-of-the-art performance across multiple benchmarks, significantly improving tracking accuracy in complex environments.
PLANET redefines multi-object tracking by embedding 3D scene geometry into query formation, leading to unprecedented accuracy in challenging scenarios.
Monocular videos record 3D scenes as sequences of 2D image-plane projections, obscuring depth and spatial relationships. Multi-object trackers localize and associate objects primarily using appearance and geometry observed only in the image plane, inheriting these ambiguities. To address this limitation, we introduce PLANET, an end-to-end multi-object tracker designed to move beyond the image plane. As an enabling step, we lift existing 2D tracking datasets into 3D. We then form world-grounded queries by embedding reconstructed 3D scene geometry into the features and positional encodings used during query formation. An auxiliary 3D location prediction task further encourages the queries to encode object positions during training. A complementary dual-resolution temporal memory preserves this evidence across longer temporal gaps. As a result, PLANET achieves state-of-the-art performance across three diverse benchmarks.
