Search papers, labs, and topics across Lattice.
The paper introduces MORE-R1, a novel framework for Multimodal Object-Entity Relation Extraction (MORE) that leverages Large Vision-Language Models (LVLMs) and explicit stepwise reasoning. It employs a two-stage training process: supervised fine-tuning (SFT) on an automatically constructed stepwise reasoning dataset, followed by reinforcement learning (RL) using Group Relative Policy Optimization (GRPO) with a progressive sample-mixing strategy. Experiments on the MORE benchmark show that MORE-R1 achieves state-of-the-art performance, demonstrating the effectiveness of stepwise reasoning and RL for this task.
Explicitly teaching LVLMs to reason step-by-step with reinforcement learning unlocks state-of-the-art performance on multimodal object-entity relation extraction.
Multimodal Object-Entity Relation Extraction (MORE) is a challenging task in information extraction research. It aims to identify relations between visual objects and textual entities, requiring complex multimodal understanding and cross-modal reasoning abilities. Existing methods, mainly classification-based or generation-based without reasoning, struggle to handle complex extraction scenarios in the MORE task and suffer from limited scalability and intermediate reasoning transparency. To address these challenges, we propose MORE-R1, a novel model that introduces explicit stepwise reasoning with Reinforcement Learning (RL) to enable Large Vision-Language Model (LVLM) to address the MORE task effectively. MORE-R1 integrates a two-stage training process, including an initial cold-start training stage with Supervised Fine-Tuning (SFT) and a subsequent RL stage for reasoning ability optimization. In the initial stage, we design an efficient way to automatically construct a high-quality SFT dataset containing fine-grained stepwise reasoning tailored to the MORE task, enabling the model to learn an effective reasoning paradigm. In the subsequent stage, we employ the Group Relative Policy Optimization (GRPO) RL algorithm with a Progressive Sample-Mixing Strategy to stabilize training and further enhance model's reasoning ability on hard samples. Comprehensive experiments on the MORE benchmark demonstrate that MORE-R1 achieves state-of-the-art performance with significant improvement over baselines.
