Search papers, labs, and topics across Lattice.
This paper introduces Abstain-R1, a 3B model trained with a novel clarification-aware RLVR reward to improve abstention and post-refusal clarification on unanswerable queries. The RLVR reward jointly optimizes for correct answers on answerable queries, explicit abstention on unanswerable queries, and semantic alignment of post-refusal clarifications. Experiments demonstrate that Abstain-R1 achieves performance competitive with larger models like DeepSeek-R1 on unanswerable queries, suggesting that calibrated abstention and clarification can be learned through verifiable rewards.
Forget scaling laws: verifiable RL can teach smaller models to abstain and clarify unanswerable questions as well as much larger models.
Reinforcement fine-tuning improves the reasoning ability of large language models, but it can also encourage them to answer unanswerable queries by guessing or hallucinating missing information. Existing abstention methods either train models to produce generic refusals or encourage follow-up clarifications without verifying whether those clarifications identify the key missing information. We study queries that are clear in meaning but cannot be reliably resolved from the given information, and argue that a reliable model should not only abstain, but also explain what is missing. We propose a clarification-aware RLVR reward that, while rewarding correct answers on answerable queries, jointly optimizes explicit abstention and semantically aligned post-refusal clarification on unanswerable queries. Using this reward, we train Abstain-R1, a 3B model that improves abstention and clarification on unanswerable queries while preserving strong performance on answerable ones. Experiments on Abstain-Test, Abstain-QA, and SelfAware show that Abstain-R1 substantially improves over its base model and achieves unanswerable-query behavior competitive with larger systems including DeepSeek-R1, suggesting that calibrated abstention and clarification can be learned through verifiable rewards rather than emerging from scale alone.
