Search papers, labs, and topics across Lattice.
This paper investigates the compositional generalization capabilities of object-centric (OC) vs. dense image representations in visual question answering across three datasets: CLEVRTex, Super-CLEVR, and MOVi-C. Using DINOv2 and SigLIP2 as dense encoders and their OC counterparts, the authors systematically control for training data diversity, sample size, representation size, and downstream compute. The key finding is that OC representations outperform dense representations in harder compositional generalization settings, are more sample efficient, and are superior when dataset size, training data diversity, or downstream compute are constrained.
Object-centric representations win at compositional generalization when data is scarce, diverse, or compute-constrained, challenging the supremacy of dense representations in visually rich settings.
Compositional generalization, the ability to reason about novel combinations of familiar concepts, is fundamental to human cognition and a critical challenge for machine learning. Object-centric (OC) representations, which encode a scene as a set of objects, are often argued to support such generalization, but systematic evidence in visually rich settings is limited. We introduce a Visual Question Answering benchmark across three controlled visual worlds (CLEVRTex, Super-CLEVR, and MOVi-C) to measure how well vision encoders, with and without object-centric biases, generalize to unseen combinations of object properties. To ensure a fair and comprehensive comparison, we carefully account for training data diversity, sample size, representation size, downstream model capacity, and compute. We use DINOv2 and SigLIP2, two widely used vision encoders, as the foundation models and their OC counterparts. Our key findings reveal that (1) OC approaches are superior in harder compositional generalization settings; (2) original dense representations surpass OC only on easier settings and typically require substantially more downstream compute; and (3) OC models are more sample efficient, achieving stronger generalization with fewer images, whereas dense encoders catch up or surpass them only with sufficient data and diversity. Overall, object-centric representations offer stronger compositional generalization when any one of dataset size, training data diversity, or downstream compute is constrained.
