Search papers, labs, and topics across Lattice.
This study investigates how multimodal large language models (MLLMs) perform visual search compared to humans, using a foveated input approach that mirrors human eye movements. While MLLMs demonstrate superior target detection and acquisition efficiency, their gaze processes reveal significant divergences from human behavior, characterized by low-entropy, large-amplitude scanpaths. These findings suggest that while MLLMs can match human outcomes in visual tasks, they do not replicate the temporal dynamics of human visual search, indicating limitations in current attention-alignment metrics.
MLLMs can outperform humans in target detection and acquisition but fail to mimic the nuanced temporal dynamics of human visual search.
Human visual search is serial: the fovea must land on a candidate to confirm it, and those landings form a scanpath. Whether multimodal large language models (MLLMs), given the same foveated input, search as humans do bears on their use as models of human vision and on attention-alignment scores. We compare three general-purpose MLLMs with human eye-movement scanpaths on goal-directed search (COCO-Search18), driving each model fixation by fixation through an identical, human-matched foveated view and assessing it along three axes: the decision of target presence, the efficiency of reaching the target, and the gaze process itself. The axes dissociate. On the decision and on target acquisition the models match or exceed humans, detecting present targets near ceiling and reaching them on the first saccade more often than people do. The gaze process is not human. Under the human-matched condition, all three share one signature: low-entropy, large-amplitude, self-consistent scanpaths that agree with themselves far more closely than two humans agree with each other. That is consistent with a single-pass, non-serial architecture rather than a limit of acuity. Matched retinal input reproduces where humans look but not how the looking unfolds in time, and no degradation regime recovers human-like search at human-like success. The gap sits on a process axis that answer-alignment and saliency metrics do not measure. Because they miss it, such metrics cannot certify human-like vision, and zero-shot models suit outcome and spatial questions but not temporal, process-level ones.
