Search papers, labs, and topics across Lattice.
The paper identifies a surprising weakness in VLMs: accurately reading analog clocks in real-world images, despite their strong performance on other multimodal reasoning tasks. To address this, the authors introduce TickTockVQA, a new human-annotated dataset of analog clocks in diverse real-world scenarios, and Swap-DPO, a direct preference optimization fine-tuning framework. Experiments show that fine-tuning VLMs with TickTockVQA and Swap-DPO significantly improves clock reading accuracy and robustness.
VLMs, despite their prowess, struggle with a seemingly simple task: reading analog clocks in real-world images, a gap this work closes with a new dataset and fine-tuning method.
Advances in vision-language models (VLMs) have achieved remarkable success on complex multimodal reasoning tasks, leading to the assumption that they should also excel at reading analog clocks. However, contrary to this expectation, our study reveals that reading analog clocks in real-world environments remains a significant challenge for state-of-the-art VLMs. Existing analog clock datasets are largely synthetic or planar with limited stylistic diversity and minimal background context, failing to capture the visual variability of real-world scenes. As a result, VLMs trained on such data exhibit weak spatial-temporal reasoning, frequently confusing the hour and minute hands and struggling under common visual conditions such as occlusion, lighting variation, and cluttered backgrounds. To address this issue, we introduce TickTockVQA, a human-annotated dataset containing analog clocks in diverse real-world scenarios. TickTockVQA provides explicit hour and minute annotations, and includes an AM/PM tag when it is inferable from the visual context. Furthermore, we propose Swap-DPO, a direct preference optimization based fine-tuning framework to align model reasoning toward accurate time interpretation. Experimental results demonstrate that our approach substantially enhances clock reading accuracy and robustness under real-world conditions, establishing a foundation for future research on spatial-temporal reasoning and visual understanding in VLMs.
