Search papers, labs, and topics across Lattice.
SAGE learns disentangled salient factors from a target dataset by contrasting it with a background dataset within a frozen visual foundation model's latent space. This allows for unsupervised discovery of subtypes and generation of novel examples conditioned on these salient representations, achieving high-fidelity reconstruction and improved subtype accuracy. The method demonstrates efficacy across diverse domains, including image generation and medical imaging analysis, by effectively separating target-specific features from common content.
Unsupervisedly discover and generate specific subtypes of visual data, like distinct eyewear styles or retinal diseases, by disentangling salient factors from common content within foundation model latents.
Given a target dataset, such as faces with eyeglasses, and a background dataset, such as faces without, contrastive analysis separates \textit{salient} factors specific to the target from \textit{common} content shared by both. We aim for salient representations that capture target-specific detail in each image, such as the shape, color, and position of the glasses, so that they reveal subtypes without subtype labels and guide the generation of new examples of a discovered subtype, even one with no name or text description. We introduce SAGE, which learns both factors directly in the high-dimensional spatial latent of a frozen representation autoencoder and conditions a diffusion transformer on the learned salient representation of a reference image. On Digits-ImageNet and FFHQ eyeglasses, SAGE combines high-fidelity \textit{reconstruction} (rFID below $2$) with unsupervised \textit{subtype discovery}, recovering the digits better than baselines (probe accuracy $0.950$ vs.\ at most $0.281$) and revealing eyewear types, finer sunglasses styles, and mislabeled images; salient-conditioned \textit{generation} raises Digits-ImageNet subtype accuracy over the unfactorized latent ($90.5\%$ vs.\ $27.7\%$) and diversity on both datasets. On retinal OCT, SAGE's salient space separates three diseases using only normal/disease labels.
