Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces outcome-grounded confidence calibration for LLM agents, training them to predict eventual task success based on per-step confidence signals derived from multi-step episode outcomes. By applying a proper scoring rule under GRPO to success labels from rollouts, the method significantly improves the AUROC of step confidence against episode outcome in offline RL benchmarks. Online RL experiments demonstrate substantial improvements in calibration and selective execution, enabling agents to identify likely failures from the initial steps.
LLM agents can learn to predict their own failure from the very first step of a task, achieving a 4.3x higher success rate in their most confident actions.
Controlling an acting LLM agent means deciding, at each step of a long-horizon task, whether to trust its next action, abstain, or hand off to a human. That decision needs a per-step confidence signal predicting whether the current rollout will eventually succeed. The agent's own self-reported verbalized confidence is the natural candidate, but existing reinforcement learning methods calibrate it against the correctness of a decision at the current step. We propose a method to calibrate it against the eventual outcome of a multi-step episode: under GRPO, the group of rollouts for a task supplies success labels, and a proper scoring rule scores each step's confidence against the realized outcome. In offline RL on recorded trajectories of a Qwen2.5-7B agent in ALFWorld, TextWorld, and WebShop environments, the AUROC of step confidence against episode outcome rises from near chance to 0.68-0.85, and the top-10% of steps the agent is most confident about succeed ~4.3x more often than average. In online RL on the live ALFWorld agent, four calibration rewards improve early-step AUROC, expected calibration error goes from 0.42 to below 0.12, and selective execution improves over both the untrained agent and a format-only RL control, without degrading task success. We also introduce an evaluation protocol that separates episode depth, task difficulty, and attempt-level foresight, which shows where the gain lies: the trained agent recognizes from its first step which tasks it will fail, while telling a succeeding attempt from a failing one at the same task is harder, with some early-window signal emerging on TextWorld.