Search papers, labs, and topics across Lattice.
This paper introduces Imaginative Perception Tokens (IPT), which serve as intermediate perceptual representations that enhance spatial reasoning in vision language models (VLMs) by allowing them to infer unseen viewpoints and integrate partial observations. The authors evaluate IPT across three tasks鈥擯erspective Taking, Path Tracing, and Multiview Counting鈥攗sing a dataset of approximately 20,000 examples, demonstrating that IPT supervision consistently improves spatial reasoning performance compared to traditional textual chain of thought training. Notably, IPT achieves a 3.4% accuracy improvement on the Multiview Counting task and competes effectively with strong closed-source models, highlighting its potential to provide a principled supervision signal for reasoning about unobserved spatial structures.
Imaginative Perception Tokens boost spatial reasoning in VLMs, achieving a 3.4% accuracy gain on Multiview Counting while outperforming traditional training methods.
Vision language models (VLMs) excel at many tasks but still struggle with spatial reasoning when critical information is not directly observable. Many such problems require imaginative perception: inferring what would be seen from an unseen viewpoint, tracing paths through occluded spaces, or integrating partial observations into a coherent spatial representation. We introduce Imaginative Perception Tokens (IPT), intermediate perceptual representations that externalize what a VLM would perceive under alternative spatial configurations while remaining consistent with the observed input. To study this capability, we formulate three tasks, Perspective Taking (PET), Path Tracing (PT), and Multiview Counting (MVC), and construct datasets of approximately 20K examples with ground truth imaginations, answers, and evaluation benchmarks. Using the unified VLM BAGEL as the backbone, IPT supervision consistently improves spatial reasoning and often outperforms textual chain of thought training, even without generating images at inference time. On MVC, IPT improves accuracy by 3.4% and achieves competitive performance with strong closed-source models on PT. We further find that combining IPT and label-only supervision yields additional gains, whereas textual chain of thought can substantially degrade performance, suggesting a modality mismatch when spatial computation is forced through language. Overall, IPT provides a principled supervision signal for reasoning about unobserved spatial structure, improving generalization while producing interpretable intermediate representations.