Search papers, labs, and topics across Lattice.
This study investigates the alignment of spatial attention maps from six vision-language models with human fixation patterns during two tasks. The research reveals that while both encoder and decoder architectures influence attention alignment, the choice of decoder is more critical, with LSTM decoders achieving a 40-50 percentage point increase in alignment compared to Transformers. Notably, CNN-LSTM models exhibited the highest overall alignment, yet LSTM attention maps were less task-differentiated, highlighting a trade-off between alignment and spatial concentration in attention mechanisms.
LSTM decoders can boost attention alignment with human fixation patterns by up to 50 percentage points, but at the cost of spatial precision and task differentiation.
Visual perception depends on top-down goals and bottom-up sensory mechanisms. Vision-language models implement both, allowing us to treat each component as a separable hypothesis about what drives where we look. We compared spatial attention maps from six vision-language models against human fixation heatmaps recorded on 200 images during two tasks (general description and social captioning). The six models spanned a 2$\times$2 factorial of CNN vs.\ ViT encoders crossed with LSTM vs.\ Transformer decoders, plus Molmo 7B-D and Qwen3.5 9B. We found that both decoder and encoder architecture shaped alignment, but decoder choice dominated. LSTM vs.\ Transformer decoders increased alignment by 40--50 percentage points (80--87\% vs.\ 40--59\% of the human noise ceiling). In contrast, CNN vs.\ ViT encoders contributed a secondary 5--20 point advantage depending on decoder family, with CNN-LSTM the most aligned model overall (85--87\%). Despite their alignment advantage, LSTM-decoder attention maps were spatially diffuse and minimally task-differentiated; ViT-Transformer, the weakest in alignment, showed the sharpest spatial concentration and strongest task differentiation. A hemispatial-neglect simulation confirmed that ablating attention impacted LSTM decoders more than Transformer decoders. In an exploratory extension using TRIBE-simulated synthetic neural responses, fixation alignment and neural relevance dissociate: CNN-Transformer attention maps better predicted synthetic brain activity despite lower fixation alignment, with attention maps best predicting early visual cortex. Together, top-down and bottom-up components trade off what they predict in behavioral and synthetic neural data.
Mila