Search papers, labs, and topics across Lattice.
This paper addresses the limitations of traditional domain generalization (DG) methods that enforce global invariance across all source domains, which can restrict the representation space by discarding valuable predictive factors. The authors introduce a novel approach called subset-shared invariance, implemented through a mixture-of-experts architecture that allows for domain-specific alignment while maintaining a routing mechanism for prediction. Experimental results on DomainBed benchmarks show that this method significantly enhances out-of-domain generalization and robustness in the face of increasing domain heterogeneity.
Enforcing global invariance in domain generalization can lead to the loss of critical predictive factors, but subset-shared invariance allows for more nuanced representation learning that boosts performance.
Domain generalization (DG) aims to learn a model from one or more source domains that generalizes to an unseen target domain without accessing target data during training. A common approach enforces invariance of representations across all source domains, assuming predictive structure is globally shared. However, we demonstrate that enforcing invariance across more domains gradually restricts the feasible representation space, discarding transferable predictive factors that are not universally shared. To address this limitation, we propose subset-shared invariance, where predictive structure is assumed stable only within domain subsets. We implement this principle with a mixture-of-experts architecture, where each expert aligns the specific domains it serves and a routing mechanism composes subset-invariant components for prediction. This creates a routing-conditioned invariance, jointly learned with the representation. To facilitate effective decomposition, we develop training objectives that encourage selective alignment, confident and balanced routing, and diverse expert specialization. Experiments on DomainBed benchmarks demonstrate improved out-of-domain generalization and greater robustness under increasing domain heterogeneity. Our results suggest that DG should move beyond enforcing a single global invariance and instead model invariance through partially shared structure across domain subsets.
