Search papers, labs, and topics across Lattice.
This paper introduces FARCA, a novel policy optimization framework designed to enhance factual supervision in reinforcement learning by addressing the issues of noisy factual credit assignment. By decomposing the challenges into credit localization and reliability ambiguities, FARCA aligns the granularity of fact verification with policy updates, utilizing counterfactual evidence attribution to assign reliability weights to factual rewards. Experimental results demonstrate that FARCA not only improves factual accuracy in models but also maintains their general reasoning capabilities across various benchmarks.
FARCA transforms factual supervision into precise, reliability-weighted training signals, significantly boosting model factuality without sacrificing reasoning performance.
To reduce the hallucination risk caused by outcome-driven rewards in large language models trained through reinforcement learning with verifiable rewards, existing mitigation approaches introduce process-level factual supervision. However, due to coarse-grained aggregation of factual signals and the lack of reliability assessment for these signals, they create a mismatch between fact verification and policy updates. We term this noisy factual credit assignment and decompose it into two aspects: credit localization ambiguity and credit reliability ambiguity. To address these issues, we propose FARCA (Fact-Aligned Reliability-Aware Credit Assignment), a policy optimization framework that transforms factual supervision into localized, reliability-weighted token-level training signals. FARCA achieves fine-grained credit localization by aligning the granularity of fact verification with that of policy updates. It further introduces counterfactual evidence attribution, which uses the dependence of a factual judgment on key evidence as an empirical proxy for verification reliability to compute reliability weights. These weights modulate factual rewards and local policy advantages, reducing the influence of potentially unreliable signals on policy optimization. Experiments across different models and multiple factual reasoning benchmarks show that FARCA significantly improves model factuality while preserving general reasoning capabilities.
