Search papers, labs, and topics across Lattice.
This paper introduces a two-stage framework that leverages generative randomization and Cross-Variant Self-Supervised Learning to mitigate the impact of spurious correlations in deep neural networks. By generating context-shifted variants of objects while preserving their identity, the approach encourages the model to learn background-invariant representations. The method achieves state-of-the-art performance on distribution-shift benchmarks, significantly improving worst-group accuracy across multiple datasets.
Generative randomization can effectively break spurious correlations, leading to a 92.5% worst-group accuracy on challenging distribution-shift tasks.
Deep neural networks trained with Empirical Risk Minimization (ERM) often fail under distribution shifts because they exploit spurious correlations between object labels and background context. Recent generative approaches address this issue by creating counterfactual images with altered contexts, but typically use these samples as standard data augmentation, leaving the model free to retain background-sensitive representations. We propose a two-stage framework that uses generative intervention to explicitly learn background-invariant visual representations. First, we isolate the foreground object using zero-shot segmentation and generate context-shifted variants with a structure-preserving diffusion model, preserving object identity while varying the surrounding environment. We then introduce Cross-Variant Self-Supervised Learning, where variants of the same object under different backgrounds form positive pairs in a contrastive objective. This encourages the encoder to align object-centric representations while suppressing background-specific cues. Then, we fine-tune the pretrained encoder using an ERM warm-up followed by GroupDRO with layer-wise learning rates. Experiments on distribution-shift benchmarks demonstrate best worst-group performance, achieving 92.5% on Waterbirds, 81.7% on MetaShift, and 87.4% on NICO++. Code: https://github.com/surajyadav-research/GRSSL
