Search papers, labs, and topics across Lattice.
This paper introduces UNIEGO, a unified egocentric video encoder that leverages a hierarchical multi-teacher distillation framework to integrate knowledge from diverse modalities and viewpoints. By employing representation-specific Proxy models to mediate the distillation process, UNIEGO effectively translates heterogeneous teacher knowledge into a cohesive egocentric representation, enhancing the model's performance. The approach achieves state-of-the-art results in action recognition, video retrieval, and action segmentation across multiple benchmarks, illustrating the advantages of structured knowledge transfer in egocentric video understanding.
UNIEGO's innovative use of Proxy models allows it to outperform traditional multi-teacher distillation methods, achieving state-of-the-art results in egocentric video tasks.
Egocentric video understanding is inherently limited by the narrow perspective of wearable cameras: a single viewpoint, a single modality, a single model cannot capture the full richness of human action. We argue that a truly expressive egocentric representation must subsume complementary knowledge across viewpoints, modalities, and foundation model representations, yet remain deployable from egocentric video alone. To this end, we introduce a hierarchical multi-teacher distillation framework that produces UNIEGO, a unified egocentric encoder trained with nine teachers spanning ego-exo viewpoints, RGB, depth, and skeleton modalities, and four foundation models. Rather than distilling directly from heterogeneous teachers whose incompatible architectures and feature geometries induce conflicting gradients, our framework interposes a layer of representation-specific Proxy models that translate diverse teacher knowledge into a homogeneous egocentric space. A second distillation stage, Selective Proxy Distillation (SPD), then adaptively selects, for each training sample, the subset of proxies that are both correct and confident, distilling exclusively from reliable supervision and suppressing erroneous signals. SPD is further stabilized by initializing UNIEGO as a learned convex combination of proxy parameters, placing the unified model in a well-conditioned region of the loss landscape before distillation begins. UNIEGO achieves state-of-the-art performance across three egocentric video understanding tasks - action recognition, video retrieval, and action segmentation on three challenging ego-exo benchmarks, outperforming naive multi-teacher distillation baselines and demonstrating that structured, proxy-mediated knowledge transfer yields richer and more discriminative egocentric representations.
