Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
This paper investigates the impact of reporting bias in VLM training data on reasoning capabilities, focusing on spatial, temporal, negation, and counting skills. Through pragmatic analysis of OpenCLIP, LLaVA-1.5, and Molmo training data, the authors find that reporting bias leads to insufficient representation of these reasoning skills. They demonstrate that scaling data and model size does not overcome this bias, but targeted annotations can improve performance on curated benchmarks.
Scaling VLMs won't magically unlock reasoning skills; you need to address the reporting bias in training data that suppresses tacit information.
The lack of reasoning capabilities in Vision-Language Models (VLMs) has remained at the forefront of research discourse. We posit that this behavior stems from a reporting bias in their training data. That is, how people communicate about visual content by default omits tacit information needed to supervise some types of reasoning; e.g.,"at the game today!"is a more likely caption than"a photo of 37 people standing behind a field". We investigate the data underlying the popular VLMs OpenCLIP, LLaVA-1.5 and Molmo through the lens of theories from pragmatics, and find that reporting bias results in insufficient representation of four reasoning skills (spatial, temporal, negation, and counting), despite the corpora being of web-scale, and/or synthetically generated. With a set of curated benchmarks, we demonstrate that: (i) VLMs perform poorly on the aforementioned types of reasoning suppressed in the training data by reporting bias; (ii) contrary to popular belief, scaling data size, model size, and to multiple languages does not result in emergence of these skills by default; but, promisingly, (iii) incorporating annotations specifically collected to obtain tacit information is effective. Our findings highlight the need for more intentional training data curation methods, rather than counting on scale for emergence of reasoning capabilities.
AI2
OpenAI