Search papers, labs, and topics across Lattice.
This paper introduces MMLDSum-LLM, a two-stage training framework designed for multimodal long-document summarization that addresses issues of attention drift and inter-modal alignment. By employing a combination of visual-alignment weighted loss and keyword-aware weighted loss, the model achieves superior performance in summarizing complex documents that integrate both text and images. Extensive evaluations on the newly proposed MMLDSum-Bench benchmark reveal significant improvements in key information coverage and cross-modal consistency compared to existing multimodal models.
MMLDSum-LLM outperforms existing models by significantly enhancing key information coverage and cross-modal consistency in long-document summarization.
Multimodal long documents are core carriers of professional knowledge, where critical evidence is sparsely distributed across paragraphs and modalities. This easily causes key information omission and cross-modal hallucinations in summarization by multimodal LLMs. These issues stem from attention drift in long-range dependency modeling and gaps in inter-modal alignment. To address this, we introduce MMLDSum-Bench, a high-quality benchmark for multimodal long-document summarization, covering multiple domains, context-length scales, and visual-textual modality distributions. We further propose MMLDSum-LLM, a reproducible two-stage training framework that combines supervised fine-tuning with visual-alignment weighted loss and keyword-aware weighted loss, followed by GRPO with a multi-objective reward (keyword coverage, image-text alignment, ROUGE, and length control). Extensive experiments on MMLDSum-Bench, comparing against leading closed-source and open-source multimodal models under a unified evaluation protocol - including LLM-as-a-judge scoring, atomic-claim precision/recall, image-text alignment (ITA), and ROUGE - demonstrate that our approach significantly improves key-information coverage and cross-modal consistency.
