Search papers, labs, and topics across Lattice.
This paper introduces VEGAS, a novel metric for evaluating video captions by aligning them with human gaze patterns, thereby addressing the common shortcoming of existing models in capturing viewer attention. By leveraging gaze data during test time, VEGAS selects captions that better reflect individual viewer focus without requiring additional training of the model. Experimental results indicate that captions chosen using VEGAS not only align more closely with human attention but also enhance performance in caption-to-video retrieval tasks.
Captions selected with VEGAS align significantly better with human attention, boosting retrieval performance and challenging the status quo of video captioning metrics.
Vision-language models excel at video captioning, yet typically generate descriptions that fail to capture individual viewers'attention. We propose VEGAS (Video caption Evaluation via GAze Score), a training-free metric that leverages test-time gaze to sample personalized, attention-aligned text. It is a cross-modal, information-theoretic metric that quantifies how well a candidate caption matches a viewer's focus. To evaluate VEGAS, we curate a dataset of egocentric activities and instructional slides paired with synchronized gaze and reference annotations. We then select captions based on VEGAS via rejection sampling without model retraining. Experiments show that VEGAS-selected captions align significantly better with human focus and improve downstream caption-to-video retrieval, demonstrating the practical utility of incorporating viewer attention during inference.
