Search papers, labs, and topics across Lattice.
This paper investigates the routing of OCR information within Vision-Language Models (VLMs) by comparing model activations on original images versus text-inpainted versions across three architectures: Qwen3-VL, Phi-4, and InternVL3.5. Using causal interventions and principal component analysis (PCA), the authors identify architecture-specific OCR bottlenecks, finding that DeepStack models (Qwen) exhibit peak sensitivity at mid-depth, while single-stage projection models (Phi-4, InternVL) show early-layer sensitivity. The study also reveals that OCR signals are low-dimensional and transferable across datasets, and that removing OCR can paradoxically improve counting performance in certain models.
Removing OCR from certain vision-language models can *improve* counting accuracy, revealing surprising interference between OCR and other visual processing pathways.
Vision-language models (VLMs) can read text from images, but where does this optical character recognition (OCR) information enter the language processing stream? We investigate the OCR routing mechanism across three architecture families (Qwen3-VL, Phi-4, InternVL3.5) using causal interventions. By computing activation differences between original images and text-inpainted versions, we identify architecture-specific OCR bottlenecks whose dominant location depends on the vision-language integration strategy: DeepStack models (Qwen) show peak sensitivity at mid-depth (about 50%) for scene text, while single-stage projection models (Phi-4, InternVL) peak at early layers (6-25%), though the exact layer of maximum effect varies across datasets. The OCR signal is remarkably low-dimensional: PC1 captures 72.9% of variance. Crucially, principal component analysis (PCA) directions learned on one dataset transfer to others, demonstrating shared text-processing pathways. Surprisingly, in models with modular OCR circuits (notably Qwen3-VL-4B), OCR removal can improve counting performance (up to +6.9 percentage points), suggesting OCR interferes with other visual processing in sufficiently modular architectures.
