Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the complex task of translating degraded Han-Nom manuscripts into modern Vietnamese by introducing a multimodal RLHF preference-alignment framework that integrates visual and textual features. By leveraging a combination of advanced models, including CLIP for visual representation and various language models for Han-Nom and Vietnamese, the authors achieve significant improvements in translation quality metrics. The results show that the DPO method outperforms others in BLEU-4 and semantic similarity, highlighting the effectiveness of multimodal preference optimization in enhancing translations of low-resource historical texts.
Multimodal preference optimization can dramatically enhance the translation quality of degraded historical manuscripts, achieving superior results in lexical and semantic accuracy.
Translating Han-Nom manuscripts into modern Vietnamese is challenging because historical pages are often degraded, the script contains rare logographic characters, and parallel supervision is limited. We propose a multimodal RLHF preference-alignment framework that conditions Vietnamese generation on manuscript images and aligned Han-Nom source text. The model combines four streams: CLIP ViT-L/14@336 for visual features, bert-base-chinese for Han-Nom representations, vinai/phobert-base for Vietnamese representations, and T5-small encoder states. Modality-specific projections and a fusion block compress the resulting 2,048-dimensional concatenation into a shared 512-dimensional representation. Starting from the same supervised fine-tuned policy, we compare PPO, DPO, and KTO under matched work-level macro-averaged evaluation. DPO achieves the best BLEU-4, ROUGE-L, BERTScore, semantic similarity, CER, WER, and token accuracy, whereas PPO obtains the highest precision, recall, and F1. KTO remains competitive through its desirable-undesirable utility objective. All preference-aligned policies improve the BLEU-4 and semantic-similarity scores available for the SFT baseline. These results indicate that multimodal preference optimization complements supervised learning by improving lexical and semantic quality in low-resource historical translation.