Search papers, labs, and topics across Lattice.
This paper introduces REFACT, an adaptive fact-restatement framework designed to enhance long-form reasoning in large language models by ensuring contextual grounding at the appropriate granularity. By employing a two-stage supervised fine-tuning to reinforcement learning (SFT-to-RL) approach, REFACT optimizes citation utility to produce well-formed, source-traceable, and answer-sufficient facts. Experimental results demonstrate that REFACT significantly improves performance on long-context question answering and counterfactual faithfulness while reducing token consumption and preserving more relevant evidence in reasoning traces.
REFACT reduces token consumption while enhancing the density and faithfulness of reasoning traces in large language models, ensuring that every cited fact meaningfully supports the answer.
Large language models increasingly rely on long-form reasoning for complex tasks, yet their reasoning traces may drift away from the supplied context when evidence is sparse, noisy, or in conflict with parametric knowledge. Existing grounding methods either attach citations after generation or encourage evidence retrieval inside the trace, but they often do not ensure that cited content is sufficient for the local inference and final answer. We propose REFACT, an adaptive fact-restatement citation framework that trains models to decide when a reasoning step needs contextual grounding and at what granularity source facts should be restated. This design avoids both unsupported inference and indiscriminate fact copying by turning citations into answer-supporting intermediate states. REFACT is optimized with a two-stage SFT-to-RL pipeline in which a citation-utility reward encourages cited facts to be well-formed, source-traceable, and answer-sufficient. Experiments on LongBench, LV-Eval, and ConFiQA show that REFACT improves long-context QA and counterfactual faithfulness while substantially reducing token consumption. Further analysis shows that REFACT preserves more answer-bearing evidence with fewer restated facts, yielding reasoning traces that are denser rather than longer. All code and data are available at https://github.com/NEUIR/REFACT.
Tsinghua AI