Search papers, labs, and topics across Lattice.
This study investigates the texture perception capabilities of Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) by comparing their internal representations with human psychophysical data. The authors find that while ViTs exhibit aligned representations across varying texture complexities, CNNs do not align with human texture recognition performance. These results indicate that ViTs may provide a more accurate model of human texture processing than CNNs, highlighting the influence of network architecture on visual representation.
Vision Transformers outperform CNNs in modeling human texture perception, revealing a fundamental shift in how we understand visual processing in AI.
In computational vision science, Convolutional Neural Networks (CNNs) have emerged as a popular model of biological vision because of the alignment they can exhibit with neural and behavioral data in humans and animals. However, it remains unclear to what extent this alignment persists for visual tasks that extend beyond the canonical object recognition paradigm based on well defined semantic content. In this study, we diverge from the common object-centric view by focusing on another aspect of vision: texture perception. We consider textures of different complexity generated with three different algorithms from the same source images. Using a rank-based statistic, we quantify the information encoded in the internal representations of a CNN and three Vision Transformers (ViTs), and we compare the similarity of these representations to those inferred from human psychophysics data. We find that the representation of textures is aligned in different ViTs, but not between the ViTs and the CNN; that ViTs form similar representations for textures of different complexity; that human performance in recognizing textures can be better predicted from ViTs representations rather than CNN representations. Taken together, these results suggest that ViTs may capture more faithfully than CNNs how texture patterns are visually processed by humans, and that the representations of texture stimuli in computational models may be driven by the network architecture.
