Search papers, labs, and topics across Lattice.
REDIPO, a novel offline DPO pipeline, is introduced to recover diverse and valid answer modes in LLMs post-training, addressing the common issue of narrowed output spaces. It constructs preference pairs by sampling responses from both base and instruct models, rewriting base-model responses, and filtering for safety and quality, favoring marginally diverse responses. Experiments on Qwen3-4B, OLMo-3-7B, and LLaMA-3.1-8B demonstrate significant improvements in diversity metrics (NoveltyBench) while largely maintaining performance on alignment benchmarks and reducing harm.
LLMs don't have to be boring: REDIPO unlocks a 134% boost in response diversity without sacrificing alignment, proving you can teach old models new tricks.
Many open-ended instructions have multiple valid answers that users can benefit from seeing, but post-training often narrows an LLM's output space toward a small set of canonical responses. We introduce REDIPO, an offline DPO data-construction pipeline for recovering distinct valid answer modes while preserving the alignment benefits of the instruct model. For each prompt, REDIPO samples responses from both base and instruct models, rewrites base-model responses with the instruct model, filters candidates for safety and instruction-following quality, and builds preference pairs that favor marginally diverse responses among candidates with similar instruction-following reward. Across Qwen3-4B, OLMo-3-7B, and LLaMA-3.1-8B, REDIPO improves NoveltyBench distinct_k by 134%, 33%, and 44% relative to the instruct checkpoints, while DivPO changes diversity by 0%, -6%, and -4% on the same models. These gains largely maintain MTBench, IFEval, and Arena-Hard performance, and reduce direct-category HarmBench attack success rate. Ablations show that marginal-diversity pair selection and base-response rewriting drive the diversity gains, while filtering and quality-bounded pairing help maintain alignment. Overall, our results show that diverse valid answers from base-model generations can be reintroduced through carefully constructed preference data while retaining the alignment benefits of post-training. We release our code and data at https://github.com/vsamuel2003/RiDiPO.
