Search papers, labs, and topics across Lattice.
This paper introduces Diverse Scenes for Gaze Object Prediction (DiSG), a benchmark designed to evaluate Open-Vocabulary Gaze Object Prediction (OVGOP) by incorporating 86 in-the-wild categories, addressing the limitations of existing closed-vocabulary methods. The authors propose a framework that utilizes text-driven object discovery and a gaze-guided selection module to accurately localize and recognize gaze targets, even in scenarios with unseen categories. Experimental results show that the proposed model not only excels in open-vocabulary settings but also surpasses traditional methods in closed-vocabulary contexts, highlighting its versatility and effectiveness.
Open-Vocabulary Gaze Object Prediction can now effectively recognize and localize objects from a diverse array of unseen categories, outperforming traditional methods.
Gaze Object Prediction (GOP) aims to localize and recognize the objects humans attend to, a task crucial for understanding human-centric interactions. However, existing methods are typically trained under a closed-vocabulary paradigm with a fixed label space and evaluated on scene-specific datasets, limiting their applicability to real-world scenarios where gaze targets often follow a long-tail distribution or belong to unseen categories. To address this gap, we introduce Diverse Scenes for Gaze object prediction (DiSG), a benchmark containing 86 in-the-wild categories that facilitates the evaluation of Open-Vocabulary GOP (OVGOP). Building on DiSG, we propose a framework that leverages text-driven object discovery to localize potential gaze candidates, with a gaze-guided selection module to pinpoint the intended target from the candidate objects. Furthermore, to better capture semantic knowledge across diverse in-the-wild categories, we introduce Gradient-Informed Selection Tuning (GIST) to selectively update parameters most relevant to a given class vocabulary. Extensive experiments demonstrate that our proposed model performs effectively in open-vocabulary settings and also outperforms existing methods in the conventional closed-vocabulary setting. The benchmark and code is available at https://github.com/sensniu/ovgop.
DAMO