Search papers, labs, and topics across Lattice.
This paper identifies that Vision Foundation Models (VFMs) excel at scene understanding for gaze following but underperform in gaze reasoning, leading to reliance on salient objects rather than actual gaze cues. To address this, they introduce a novel training mechanism involving a head-conditioned local LoRA for localized adaptation and an out-of-cone penalty to inject gaze cues into head tokens. Experiments on GazeFollow and VAT datasets demonstrate state-of-the-art performance, especially when gaze targets lack semantic salience.
VFMs may be great at seeing, but they're surprisingly bad at *looking* – until now.
Gaze following requires both scene understanding and gaze reasoning to localize the gaze target of an in-scene person. Recently, vision foundation models (VFMs) have demonstrated strong performance on this task, enabling simpler architectures while outperforming prior methods. However, we observe a key limitation of VFM-based approaches: while VFMs substantially improve scene understanding, they contribute little to gaze reasoning. As a result, existing methods often rely on semantically salient objects rather than true gaze cues, leading to degraded performance when targets are not salient. To address this, we propose a novel training mechanism to enhance gaze reasoning in VFMs for gaze following. Our method includes: (1) a head-conditioned local LoRA, which enables localized adaptation to preserve scene token learning while improving head token learning for gaze reasoning; and (2) an out-of-cone penalty, which injects gaze cues into head tokens while aligning them with scene tokens. Experiments on the GazeFollow and VAT datasets demonstrate that our method achieves state-of-the-art performance, with particularly strong improvements when gaze targets are not semantically salient. Our findings offer valuable insights for advancing future gaze following research. We will release the code once the paper is accepted.
