Search papers, labs, and topics across Lattice.
This paper addresses the challenge of cross-path inconsistency in Vision-Text Compression (VTC), where visual representations diverge from their textual semantics due to the pretraining of vision encoders on natural images. The authors introduce SPIRAL, a self-supervised alignment framework that leverages the model's own text-path behavior for alignment, achieving significant improvements in VTC performance. Experimental results show that SPIRAL enhances the Qwen3-VL-8B model's score from 35.10 to 54.02 on VTCBench, closely approaching native text-input performance and outperforming much larger models.
SPIRAL aligns visual representations with textual semantics, achieving near-native performance in Vision-Text Compression without external supervision.
Vision-Text Compression (VTC) renders long texts into images and encodes them through the vision encoder (ViT), compressing thousands of text tokens into far fewer visual tokens. However, since the ViT is pretrained predominantly on natural images, it captures visual attributes (glyphs, font sizes, layout) rather than linguistic semantics, causing rendered-image representations to diverge from native-text representations. We term this cross-path inconsistency and show, via rendering perturbation experiments, that it is a critical yet overlooked bottleneck of VTC. We propose SPIRAL (Self-improving Path Integration and Realignment), a self-supervised alignment framework that closes this gap using only the model's own text-path behavior as supervision, requiring no external teachers or additional annotations. SPIRAL operates at two complementary granularities: token-level on-policy distillation (OPD) for local faithfulness, and sequence-level preference optimization (DPO) for global coherence. On VTCBench, SPIRAL improves the overall score of Qwen3-VL-8B from 35.10 to 54.02, approaching the native text-input performance (55.60) and outperforming models up to 30x larger. The two granularities exhibit complementary strengths: OPD excels at retrieval and is sample-efficient, while DPO is stronger on reasoning and memory and scales better with data. SPIRAL's benefits also generalize to out-of-domain benchmarks, confirming that effective VTC hinges on aligning rendered-image representations back to native-text semantics.
NUS