Search papers, labs, and topics across Lattice.
The paper investigates whether fine-tuning Vision Transformer (ViT) self-attention weights on human saliency fixation maps can improve cognitive alignment between ViTs and human attentional characteristics. They compare the fine-tuned ViT against a shuffled control to isolate semantically relevant signals. Results show significant improvements in alignment across five saliency metrics, inducing human-like biases, without sacrificing classification performance on ImageNet, ImageNet-C, and ObjectNet.
Surprisingly, ViTs can be made more human-like in their attention patterns, for free, simply by fine-tuning on human eye-tracking data, without hurting accuracy.
For state-of-the-art image understanding, Vision Transformers (ViTs) have become the standard architecture but their processing diverges substantially from human attentional characteristics. We investigate whether this cognitive gap can be shrunk by fine-tuning the self-attention weights of Google's ViT-B/16 on human saliency fixation maps. To isolate the effects of semantically relevant signals from generic human supervision, the tuned model is compared against a shuffled control. Fine-tuning significantly improved alignment across five saliency metrics and induced three hallmark human-like biases: tuning reversed the baseline's anti-human large-object bias toward small-objects, amplified the animacy preference and diminished extreme attention entropy. Bayesian parity analysis provides decisive to very-strong evidence that this cognitive alignment comes at no cost to the model's original classification performance on in- (ImageNet), corrupted (ImageNet-C) and out-of-distribution (ObjectNet) benchmarks. An equivalent procedure applied to a ResNet-50 Convolutional Neural Network (CNN) instead degraded both alignment and accuracy, suggesting that the ViT's modular self-attention mechanism is uniquely suited for dissociating spatial priority from representational logic. These findings demonstrate that biologically grounded priors can be instilled as a free emergent property of human-aligned attention, to improve transformer interpretability.
