Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To investigate whether visual representations capture abstract conceptual knowledge rather than merely low-level perceptual features, the authors probed vision encoders and vision-language models (VLMs) for canonical object colors across paired RGB and grayscale images. They find that canonical colors remain linearly decodable from grayscale representations and correlate directly with predicted object identities, while VLM post-training significantly alters these decodable visual semantics. These findings demonstrate that visual representations internalize high-level conceptual priors independent of surface-level cues, providing a controlled lens for evaluating representational drift during multimodal alignment.
Vision encoders encode an object's canonical color even when all color is stripped from the input image, exposing semantic priors that VLM post-training radically reorganizes.
Visual encoders construct a representation of the image input for Vision-Language models. How much conceptual, as opposed to immediately visible, information does this representation contain? We use canonical color as a controlled test case to ask whether vision encoders make canonical-color information linearly accessible, even when color is removed from the input image. We construct a dataset of objects with canonical colors, and probe vision encoders for both color and object identity using color and grayscale images. We find that canonical color remains decodable from grayscale images, and is tied to predicted object identity, indicating a conceptual link. Extending this analysis to full VLMs, we find that VLM post-training can have a surprisingly large effect on color decodability in the vision encoder. Overall, canonical color provides a usefully controllable lens for tracing object-level conceptual semantic information in vision encoders and VLMs.