Search papers, labs, and topics across Lattice.
This paper introduces OmniRef-Bench, a benchmark designed to evaluate multi-reference image generation (MRIG) tasks that involve complex combinations of reference image types. The authors propose DyRef, a two-stage training framework that incorporates Difficulty-aware Advantage Reweighting (DAR) and Discriminative Reward Scaling (DRS) to enhance model performance in MRIG scenarios. Experimental results reveal that DyRef substantially improves the performance of mainstream open-source models on OmniRef-Bench, particularly as the complexity of reference images increases.
Mainstream models falter in multi-reference image generation, but DyRef's innovative training framework boosts their performance significantly.
While personalized image generation has achieved remarkable progress, multi-reference image generation (MRIG) remains a challenging task. Most existing benchmarks fail to adequately evaluate complex MRIG scenarios, hindering further progress in this area. To better assess model performance on complex MRIG tasks, we introduce OmniRef-Bench, a benchmark that covers complex combinations of reference image types and a large number of reference images. Evaluations on OmniRef-Bench show that mainstream open-source models struggle in complex MRIG scenarios, and their performance deteriorates significantly as the number of mixed-type reference images increases. To address this issue, we propose DyRef, a two-stage training framework. In the first stage, supervised fine-tuning equips the model with the basic capability to handle complex MRIG tasks. In the second stage, we introduce Difficulty-aware Advantage Reweighting (DAR) and Discriminative Reward Scaling (DRS). DAR dynamically adjusts the optimization objective to improve performance when handling a large number of mixed-type reference images. DRS enlarges intra-group reward differences for more effective policy optimization. Experiments demonstrate that DyRef significantly improves the performance of open-source models on OmniRef-Bench and single-image editing benchmarks, demonstrating the effectiveness and generalization capability of our approach.
