Search papers, labs, and topics across Lattice.
This paper introduces Shape-of-You (SoY), a novel framework for unsupervised semantic correspondence that leverages Fused Gromov-Wasserstein (FGW) optimal transport to generate pseudo-labels. SoY incorporates structural consistency by using a 3D foundation model to define intra-structure in geometric space, addressing ambiguities arising from symmetries and repetitive features in 2D images. By approximating FGW through anchor-based linearization and employing a soft-target loss, SoY achieves state-of-the-art performance on SPair-71k and AP-10k datasets, demonstrating effective semantic correspondence without explicit geometric annotations.
By fusing 3D geometric priors into Gromov-Wasserstein optimal transport, SoY unlocks state-of-the-art unsupervised semantic correspondence, even in geometrically ambiguous scenes.
Semantic correspondence is essential for handling diverse in-the-wild images lacking explicit correspondence annotations. While recent 2D foundation models offer powerful features, adapting them for unsupervised learning via nearest-neighbor pseudo-labels has key limitations: it operates locally, ignoring structural relationships, and consequently its reliance on 2D appearance fails to resolve geometric ambiguities arising from symmetries or repetitive features. In this work, we address this by reformulating pseudo-label generation as a Fused Gromov-Wasserstein (FGW) problem, which jointly optimizes inter-feature similarity and intra-structural consistency. Our framework, Shape-of-You (SoY), leverages a 3D foundation model to define this intra-structure in the geometric space, resolving abovementioned ambiguity. However, since FGW is a computationally prohibitive quadratic problem, we approximate it through anchor-based linearization. The resulting probabilistic transport plan provides a structurally consistent but noisy supervisory signal. Thus, we introduce a soft-target loss dynamically blending guidance from this plan with network predictions to build a learning framework robust to this noise. SoY achieves state-of-the-art performance on SPair-71k and AP-10k datasets, establishing a new benchmark in semantic correspondence without explicit geometric annotations. Code is available at Shape-of-You.
