Search papers, labs, and topics across Lattice.
This paper introduces Trajectory-Informed Advantage Reweighting (TIAR), a novel method for LLM abstention learning that dynamically re-weights the abstention reward during Group Relative Policy Optimization (GRPO) training. TIAR leverages GRPO's multiple trajectories as a confidence indicator to calculate the abstention advantage, encouraging consistency and exploration of knowledge boundaries. Empirical results on AbstentionBench demonstrate that TIAR achieves state-of-the-art abstention F1 scores across most evaluation categories, outperforming static ternary baselines while preserving baseline accuracy.
LLMs can learn to abstain from answering questions they're unsure about with state-of-the-art accuracy by dynamically re-weighting abstention rewards based on trajectory consistency during training.
This paper investigates large language model (LLM) abstention learning, specifically using ternary reward, which incentivize truthfulness in large language models. This paper extends that idea by moving from a ternary reward to a Trajectory-Informed advantage reweighting, dynamically re-weights the abstention reward during Group Relative Policy Optimization (GRPO) training. The objective of this work focuses on abstention learning instead of improving truthfulness, serving as an exploration into hallucination reduction. The novelty of this paper lies in methodological innovation, advantage re-weighting, and benchmark selection. Leveraging GRPO's multiple trajectories as a natural abstention signal, this method uses a reward signal to explore knowledge boundaries and encourage consistency. By demonstrating that trajectories can be used as a confidence indicator of the policy relative to the query, they are then used to dynamically calculate the abstention advantage. AbstentionBench is used as the evaluation benchmark, as this work aims to contribute to the field of abstention learning. All datasets on the benchmark were tested against this method and various baselines. Empirical results demonstrate that TIAR achieves state-of-the-art abstention F1 scores across five of six evaluation categories, outperforming the static ternary baseline on 17 of 31 benchmark datasets while fully preserving baseline accuracy.
