Search papers, labs, and topics across Lattice.
The paper introduces Grid2Matrix (G2M), a controlled benchmark where VLMs must transcribe a color grid into a numerical matrix based on a provided color-to-number mapping. Results show that VLMs exhibit a "digital agnosia," failing abruptly on relatively small grids, despite their visual encoders preserving more grid information than the final language output. Analysis reveals that this failure is structured, depending on grid cell alignment with visual patch boundaries, and is not fully addressed by model scaling or multimodal alignment.
VLMs suffer from "digital agnosia," exhibiting a surprisingly sharp failure to transcribe even small color grids into matrices, revealing a critical gap between visual feature encoding and language generation.
Vision-Language Models (VLMs) excel on many multimodal reasoning benchmarks, but these evaluations often do not require an exhaustive readout of the image and can therefore obscure failures in faithfully capturing all visual details. We introduce Grid2Matrix (G2M), a controlled benchmark in which a model is shown a color grid and a color-to-number mapping, and must output the corresponding matrix. By varying grid size and the number of colors, G2M provides a simple way to increase visual complexity while minimizing semantic confounds. We find that VLMs exhibit a sharp early collapse in zero-shot end-to-end evaluation, failing on surprisingly small grids rather than degrading gradually as the task becomes denser. We probe the visual encoders of VLMs from two representative families and find that they preserve substantially more of the grid information than the corresponding end-to-end outputs. This suggests that the failure is not explained by visual encoding alone, but also reflects a gap between what remains recoverable from visual features and what is ultimately expressed in language. We term this gap \textit{Digital Agnosia}. Further analyses show that these errors are highly structured and depend strongly on how grid cells overlap with visual patch boundaries. We also find that common strategies such as model scaling and multimodal alignment do not fully eliminate this failure mode. We expect G2M to serve as a useful testbed for understanding where and how VLMs lose fine visual details, and for evaluating tasks where missing even small visual details can matter, such as tables, charts, forms, and GUIs.
