Search papers, labs, and topics across Lattice.
This paper introduces a rubric-based reward framework for open-domain question answering that generates query-specific rubrics grounded in retrieved evidence, allowing for fine-grained supervision during post-training. By decomposing answer quality into multiple dimensions, the approach significantly enhances performance, achieving a 6.5% improvement over instruction-tuned baselines and a 4% improvement over simpler rubric variants across various evaluation axes. The findings indicate that this method not only bolsters factual support but also enhances coherence and adherence to query requirements, demonstrating the effectiveness of multi-dimensional rubrics in complex answer generation tasks.
Grounded, multi-dimensional rubrics can boost answer quality in open-domain question answering by over 6%, transforming how we evaluate and train AI systems for complex queries.
Designing effective reward signals for open-domain question answering is challenging because high-quality responses must simultaneously satisfy multiple aspects of answer quality that are difficult to capture with a holistic scalar objective. We introduce a rubric-based reward framework that generates query-specific rubrics grounded in retrieved evidence and decomposed into multiple quality dimensions, providing fine-grained supervision during post-training. Averaged across three evaluation axes (composition, grounding, and instruction-following), our approach improves over the instruction-tuned baseline by 6.5% and over flat rubric variants by 4%, with consistent gains across all evaluation datasets. Conditioning rubrics on retrieved evidence improves factual support, while decomposing rubrics into quality-specific dimensions further improves coherence, organization, and adherence to query requirements. Our results show that grounded, multi-dimensional rubrics provide more effective reward supervision for complex open-domain question answering.
