Search papers, labs, and topics across Lattice.
This paper introduces Whareformer, a novel transformer-based model designed to tackle the OSNOM task of tracking objects in long egocentric videos, even when they become occluded or leave the field of view. By integrating an updatable memory for established tracks and a feed-forward track assignment module, Whareformer effectively reasons about both object appearance and 3D location. Remarkably, despite being trained on only 56 videos, it achieves state-of-the-art performance on 260 long test videos across three datasets, demonstrating significant improvements over previous methods.
Whareformer outperforms prior models in tracking occluded objects in egocentric videos, achieving state-of-the-art results with minimal training data.
The recently established'Out of Sight, Not out of Mind'(OSNOM) task for egocentric videos focuses on tracking objects that are moved by the camera wearer, online, maintaining knowledge of instance locations throughout the video even when they leave the field of view or become heavily occluded. In this paper, we propose the first learning-based solution to the OSNOM task: Whareformer, a transformer-based model with two components: an updatable memory of established tracks and a track assignment module that associates observations with existing tracks in a feed-forward manner. Whareformer jointly reasons over evolving object appearance (what) and updated 3D location (where), and employs a dedicated New Track token to reason about novel objects. Thanks to its design choices of using relative distances and evolving track representations, Whareformer is trained on a small set of 56 videos but achieves SOTA performance on 260 long test videos from three datasets: EPIC-KITCHENS-100 (unseen videos), IT3DEgo, and HD-EPIC, with significant absolute improvements over prior work.
