Search papers, labs, and topics across Lattice.
This paper introduces CARE-PPO, a reinforcement learning framework that integrates loss prediction for uncertainty estimation with actor-critic PPO fine-tuning, allowing for the simultaneous learning of accurate numerical predictions and reliable confidence signals. By employing a Confidence-Aligned Reward for Estimation, the framework provides dense, error-aware feedback to enhance both prediction accuracy and confidence alignment. Evaluated on healthcare and finance tasks with Qwen-3 models, CARE-PPO not only improves quantitative prediction performance but also yields better-aligned confidence estimates compared to traditional methods, demonstrating robustness in out-of-distribution scenarios.
Confidence estimates from CARE-PPO significantly outperform traditional logit-based methods, ensuring that LLMs can predict not just accurately, but also reliably.
LLMs can perform language-based quantitative prediction from unstructured inputs, but remain susceptible to hallucinations and overconfident errors, making it critical to know not only what a model predicts, but when its predictions can be trusted. We introduce CARE-PPO, a reinforcement learning framework that establishes a connection between loss prediction for uncertainty estimation and actor-critic PPO fine-tuning, enabling joint learning of accurate numerical estimates and reliable confidence signals in language-based quantitative prediction. CARE-PPO uses a Confidence-Aligned Reward for Estimation, defined as a function of prediction error, to provide dense error-aware feedback to the actor while inducing the critic to learn a value function aligned with prediction quality. During inference, we repurpose the critic as a confidence estimator. Across two real-world tasks in healthcare and finance and two Qwen-3 model scales (4B and 8B), CARE-PPO achieves strong quantitative prediction performance, while producing significantly better-aligned confidence estimates through the critic than logit-based and verbalized baselines. These gains persist under realistic out-of-distribution settings across domains, spanning linguistic and domain shifts. Finally, CARE-PPO reduces task-specific overfitting on general instruction-following prompts, consistent with the broader generalization advantages of RL fine-tuning over supervised approaches.
