Search papers, labs, and topics across Lattice.
The paper introduces GLoTran, a global-local dual visual perception framework for MLLM-based Text Image Machine Translation (TIMT) that addresses challenges in high-resolution, text-rich images. GLoTran integrates a low-resolution global image with multi-scale region-level text image slices under an instruction-guided alignment strategy, enabling MLLMs to maintain contextual consistency and capture fine-grained textual details. The authors also construct GLoD, a large-scale text-rich TIMT dataset with 510K high-resolution global-local image-text pairs, and demonstrate that GLoTran improves translation completeness and accuracy compared to existing MLLMs.
Achieve significantly better translation of text-rich images by giving MLLMs both the big picture and close-up details.
Text Image Machine Translation (TIMT) aims to translate text embedded in images in the source-language into target-language, requiring synergistic integration of visual perception and linguistic understanding. Existing TIMT methods, whether cascaded pipelines or end-to-end multimodal large language models (MLLMs),struggle with high-resolution text-rich images due to cluttered layouts, diverse fonts, and non-textual distractions, resulting in text omission, semantic drift, and contextual inconsistency. To address these challenges, we propose GLoTran, a global-local dual visual perception framework for MLLM-based TIMT. GLoTran integrates a low-resolution global image with multi-scale region-level text image slices under an instruction-guided alignment strategy, conditioning MLLMs to maintain scene-level contextual consistency while faithfully capturing fine-grained textual details. Moreover, to realize this dual-perception paradigm, we construct GLoD, a large-scale text-rich TIMT dataset comprising 510K high-resolution global-local image-text pairs covering diverse real-world scenarios. Extensive experiments demonstrate that GLoTran substantially improves translation completeness and accuracy over state-of-the-art MLLMs, offering a new paradigm for fine-grained TIMT under high-resolution and text-rich conditions.
