Search papers, labs, and topics across Lattice.
This paper introduces Outcome Monitors, a novel mechanism that enhances agent performance by detecting violations of outcome contracts during tool failures and providing recovery options. By integrating these monitors, the authors achieved a significant increase in ToolMaze completion rates from 10.9% to 28.1% across multiple models, demonstrating their effectiveness in mitigating silent tool failures. The results indicate that the presence of recovery tools is critical for improving task completion, while the ability to detect violations outside a mined vocabulary remains a challenge.
Tool failures can be effectively managed with Outcome Monitors, boosting task completion rates by over 150% in critical scenarios.
When a tool call times out, the agent sees the failure and can route around it. A cached error page or negative price can instead arrive in the expected format and be consumed as fact. We introduce Outcome Monitors, which detect violations of outcome contracts mined from task-disjoint traces or derived from public schemas. On a violation, the monitor preserves the result and issues a nonbinding receipt naming the violated property and public recovery tools. In frozen, prespecified evaluations with injected failures, Outcome Monitors raise ToolMaze completion from 10.9% to 28.1% across four models in two provider families and replicate in a third. In tau-bench retail, completion improves by 14.0 and 12.0 points on two tiers. In separate ToolMaze controls, removing the recovery-tool list eliminates the measured gain and restoring it recovers the effect; diagnostic detail and timing produce no detectable differences. Gains concentrate where the fault blocks completion. On a suite transcribed from a published incident taxonomy, detection outside the mined vocabulary falls to 46%, though delivery continues and completion is unchanged. Recovery tools are the active receipt content in these controls; extending detection beyond the contract vocabulary remains open.
