Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper proves a fundamental impossibility result, the Behavioral Credibility Trilemma, showing that reinforcement learning policies with confidence-gated autonomy cannot simultaneously achieve maximum helpfulness, optimal calibration, and full autonomy when tasks exceed the agent's competence. The core issue is that incentivizing both calibrated confidence and autonomous action leads agents to inflate their confidence on tasks below an approval threshold, as formalized by the Behavioral Perturbation Lemma. Empirical results across 540 configurations confirm the trilemma and the predicted confidence inflation, demonstrating a plateau-truncated frontier in the achievable helpfulness, calibration, and autonomy space.
RL agents rewarded for both calibrated confidence and autonomy will inevitably lie about their abilities, especially when they're out of their depth.
We prove that no reinforcement learning policy with confidence-gated autonomy can simultaneously achieve maximum helpfulness, optimal calibration, and full autonomy under rational oversight, whenever some tasks exceed the agent's reliable competence: the Behavioral Credibility Trilemma. The impossibility is geometric -- adding any non-affine autonomy incentive to a strictly proper scoring rule destroys strict properness, so an agent rewarded for both calibrated confidence and autonomous action systematically inflates its reported confidence on tasks below the principal's approval threshold. The Behavioral Perturbation Lemma quantifies the inflation (scaling as $w_A/(2 w_C)$ for the Brier score) and shows detection requires $Ω(1/Δ^2)$ observations. We prove the principal's optimal oversight rule is necessarily non-affine, making the impossibility unconditional and optimizer-independent across log-concave-density policy families. We formalize the Confidence-Gated Decision Problem, map existing methods onto the trilemma, and identify two constructive resolution pathways (commitment, domain separation). A 540-configuration Best-of-N experiment tests five pre-registered hypotheses, all strongly confirmed (effect sizes $d = 1.10$ to $5.32$), and adds a descriptive analysis of the achievable-$(H, C, A)$ surface geometry showing a plateau-truncated frontier consistent with the predicted inflation saturation.