Search papers, labs, and topics across Lattice.
This paper introduces PRISM, a novel approach for cross-view video representation learning that effectively disentangles view-invariant and view-variant semantics by decomposing video into separate latent features. By leveraging language supervision for recomposition, PRISM addresses the entanglement issue seen in existing methods, which often fail to maintain independent semantics across different viewpoints. The method achieves state-of-the-art performance on multiple benchmarks, including EgoExo4D and EgoExoLearn, even outperforming in-domain models in zero-shot scenarios.
View-invariant video representations can be achieved without sacrificing the richness of view-variant semantics, leading to unprecedented performance in cross-view tasks.
Cross-view video representation learning aims to capture viewpoint-invariant action semantics despite substantial appearance changes across egocentric and exocentric videos. However, existing methods encode each video as a unified embedding, where view-invariant and view-variant semantics inevitably entangle under co-occurrences - a failure mode we show persists even in cross-view methods explicitly trained for view-invariance. Our key insight is that a view-invariant feature is truly disentangled when it can be sufficiently recomposed with an arbitrary view-variant feature while preserving their independent semantics. Building on this, we propose PRISM, that decomposes video into view-invariant and view-variant latents and recompose them under language supervision encouraging clean decomposition of the two streams. PRISM achieves state-of-the-art results on EgoExo4D, EgoExoLearn, AE2, even surpassing in-domain models under zero-shot setting. Code is available at https://github.com/litcoderr/prism.
