Search papers, labs, and topics across Lattice.
This paper introduces DeepGaze3.5-VL, a novel framework for modeling human visual scanpaths by framing the task as discrete sequence modeling through autoregressive token prediction. By leveraging pretrained Vision-Language Models and mapping spatial coordinates into a text vocabulary, the model allows for flexible conditioning based on viewer identities and task-specific objectives, achieving a significant 46% improvement in Information Gain over previous methods. Additionally, the framework enables controlled in-silico simulations for behavioral interventions, demonstrating its utility in recovering known oculomotor phenomena from data alone.
Achieving a 46% boost in predictive performance for visual scanpath modeling, DeepGaze3.5-VL transforms how we understand human attention dynamics.
Understanding human visual attention on a scene over time has applications in domains such as interface design and inferring cognitive states. Modeling visual scanpaths has historically relied on specialized architectures with hand-crafted priors. While these architectures can model fixation sequences, their rigid structural biases restrict easy extendability and flexible conditioning. For instance, integrating task-specific instructions or adapting to distinct viewer identities requires custom, disjoint architectural additions. We frame scanpath prediction purely as a discrete sequence modeling task. By mapping coordinates into a text vocabulary, we leverage the pretrained representations of Vision-Language Models. This framing absorbs diverse factors of variation: simple prompting allows for global conditioning, such as providing viewer identities to capture personalized biases, or task-specific objectives like visual search. The framework can also integrate per-fixation attributes, such as individual fixation durations, alongside spatial locations. The autoregressive alignment enables the scalable, exact computation of per-fixation log-likelihoods, directly equivalent to the commonly used Information Gain (IG) metric. Our model, DeepGaze3.5-VL, establishes a new state-of-the-art across multiple datasets, achieving 2.18 bits of IG on MIT1003, a 46% improvement over DeepGaze III. This advantage persists even when baselines use identical high-capacity vision encoders. Beyond predictive performance, our generative framework serves as a powerful computational tool for direct behavioral interventions, allowing for controlled in-silico simulations that would be experimentally difficult or impossible to conduct in vivo. We demonstrate this ability by performing controlled interventions on the durations of pre-saccadic fixations, recovering known oculomotor phenomena purely from data.
