Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the advantages of local, sequential vision models over traditional global models in visual reasoning tasks, focusing on visual state tracking and length generalization. The authors find that while global shortcuts can hinder generalization in vision models, recurrent policies that utilize strictly local perception significantly improve performance on tasks of varying complexity. These findings suggest that incorporating local attention mechanisms is crucial for achieving robust compositional generalization in visual reasoning systems.
Local perception in vision models could be the key to unlocking robust generalization across complex tasks, challenging the dominance of global approaches.
A striking feature of the human visual system is that it ingests visual information through a series of local foveated glimpses, rather than a single global computation. This makes human vision distinctly different from most popular computer vision models in use today, which input images globally and in a single shot. A natural question therefore is whether local, sequential vision models may provide any fundamental computational benefits in addition to being biologically more plausible than global models. In this work, we investigate this question from the perspective of visual state tracking and length generalization. Inspired by recent studies of length generalization in language models, we study the behavior of vision models trained on simple vision tasks that require the aggregation of local information across an image. Our experiments reveal that, similar to language models, vision models can learn to exploit global shortcuts and thereby fail to generalize over task length or complexity. We also show that recurrent vision policies based on strictly local perception can mitigate these failures, thereby allowing models to generalize on these tasks. Our results show that local attention may be an essential overlooked requirement for robust compositional generalization.