Search papers, labs, and topics across Lattice.
This paper introduces rDPO, a Direct Preference Optimization framework that uses instance-specific rubrics to provide fine-grained feedback for visual reasoning tasks. By creating checklist-style rubrics for each image-instruction pair, rDPO improves the quality of preference data used for training. Experiments show that rDPO significantly outperforms existing methods, achieving state-of-the-art results on public reward modeling and downstream benchmarks by better capturing nuanced visual reasoning requirements.
Fine-grained rubrics unlock significantly better visual reasoning in preference optimization, rivaling GPT-5.4 with a much smaller model.
The effectiveness of Direct Preference Optimization (DPO) depends on preference data that reflect the quality differences that matter in multimodal tasks. Existing pipelines often rely on off-policy perturbations or coarse outcome-based signals, which are not well suited to fine-grained visual reasoning. We propose rDPO, a preference optimization framework based on instance-specific rubrics. For each image-instruction pair, we create a checklist-style rubric of essential and additional criteria to score responses from any possible policies. The instruction-rubric pool is built offline and reused during the construction of on-policy data. On public reward modeling benchmarks, rubric-based prompting massively improves a 30B-A3B judge and brings it close to GPT-5.4. On public downstream benchmarks, rubric-based filtering raises the macro average to 82.69, whereas outcome-based filtering drops it to 75.82 from 81.14. When evaluating scalability on a comprehensive benchmark, rDPO achieves 61.01, markedly outperforming the style-constrained baseline (52.36) and surpassing the 59.48 base model. Together, these results show that visual preference optimization benefits from combining on-policy data construction with instance-specific criterion-level feedback.
