Search papers, labs, and topics across Lattice.
This paper tackles the challenge of bias in foundation models by introducing a novel approach that combines Confidence-guided Bias Concept Mining (CBCM) and Dual-branch Cross-projection Debiasing (DCD). CBCM identifies spurious attributes without requiring group labels, while DCD utilizes a dual-branch structure to disentangle target and spurious representations, effectively mitigating bias. The proposed method achieves state-of-the-art performance in worst group accuracy on benchmark datasets with minimal parameter tuning, highlighting its efficiency and effectiveness in debiasing tasks.
Spurious correlations in foundation models can be effectively disentangled using a dual-branch approach, achieving superior bias mitigation with minimal parameter adjustments.
Foundation models trained on biased datasets often rely on spurious correlations between target labels and non-causal attributes, resulting in poor generalization on minority groups. Bias mitigation remains challenging due to two fundamental issues. First, when group labels are unavailable, existing group-unsupervised methods typically infer spurious attributes implicitly from model behavior, making it difficult to identify spurious factors that are semantically aligned with real-world biases. Second, even with pseudo spurious supervision, most existing debiasing methods follow a single-branch design that operates within a single shared feature space, where target and spurious attributes are intrinsically entangled. To address the first challenge, we introduce Confidence-guided Bias Concept Mining (CBCM), which leverages diffusion-disentangled, semantically grounded concept representations to identify reliable spurious attributes without attribute annotations. To address the second challenge, we propose Dual-branch Cross-projection Debiasing (DCD), a prompt-tuning framework that separates target and spurious representations into two branches and explicitly removes spurious information through cross null-space projection while preserving target-relevant semantics. Extensive experiments on four benchmark datasets show that our method achieves state-of-the-art worst group accuracy among group-unsupervised approaches, while tuning at most 0.22% of the model parameters. The source code is available in the supplementary materials.
Microsoft Research