Search papers, labs, and topics across Lattice.
This paper introduces UniTranslator, a unified multimodal framework designed for end-to-end in-image machine translation (IIMT), which addresses challenges such as understanding-generation conflicts and spatial position misalignment. By implementing an Understand-Generation Alignment Module (UGAM) and a Spatial Mask Decoder (SMD), the framework ensures semantic consistency and improved spatial grounding in the translation process. Extensive experiments demonstrate that UniTranslator outperforms existing methods across various benchmarks, showcasing the benefits of integrating translation understanding with image generation.
Achieving state-of-the-art performance in in-image machine translation, UniTranslator reveals a powerful synergy between translation understanding and image generation.
In-Image Machine Translation (IIMT) aims to translate scene text in an image and render the translated text back into the original regions while preserving the overall visual appearance. Recent unified multimodal models provide a promising solution by combining visual-text understanding and image generation within a single framework. However, directly adapting such models to IIMT remains challenging. In particular, they often suffer from understanding-generation conflicts, where the translation inferred during understanding is inconsistent with the text supervision used in generation, and spatial position misalignment, where the rendered text does not accurately match the target text regions. To address these issues, we present UniTranslator, a unified multimodal framework for IIMT that tightly couples translation understanding and text editing. Specifically, we introduce an Understand-Generation Alignment Module (UGAM) to bridge the representation gap between understanding and generation, encouraging semantic consistency between translated content prediction and text rendering. We further propose a Spatial Mask Decoder (SMD) with pixel-level supervision over text regions to improve spatial grounding, geometric alignment, and layout controllability during generation. Extensive experiments on multiple benchmarks demonstrate that UniTranslator achieves state-of-the-art performance across diverse language directions and complex real-world layouts. Moreover, our results reveal a strong mutual reinforcement effect between translation understanding and image generation, highlighting the advantage of unified translation multimodal learning. Code is available at https://github.com/SeerRay-Lab/Unitranslator.
