Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Calibrated Collective Oversight (CCO), a framework for scalable oversight of autonomous agents that aggregates diverse scoring functions into a penalty term based on a conservative baseline. CCO leverages Conformal Decision Theory to calibrate the level of conservatism online, providing finite-time guarantees on undesirable outcome rates without distributional assumptions. Experiments on SWE-bench and MACHIAVELLI demonstrate that CCO effectively constrains misaligned agents and reduces ethical violations while adhering to user-specified target violation thresholds.
Achieve scalable oversight of AI agents with statistical guarantees by calibrating conservatism using diverse human feedback, ensuring undesirable outcomes stay below user-defined thresholds.
Agentic AI systems capable of autonomous planning and extended environmental interaction pose a fundamental control problem: how can humans maintain meaningful oversight of systems that may exceed their own capabilities? Existing approaches to scalable oversight rely on complex assumptions, remain largely heuristic, or lack practical methods for sequential settings with statistical guarantees. We introduce Calibrated Collective Oversight (CCO), which aggregates diverse auxiliary scoring functions into a penalty measuring deviation from a conservative baseline. Inspired by Attainable Utility Preservation, CCO enables collective conservatism: actions face a penalty proportional to overseer concern, so high-utility actions are still selected when overseers find them unobjectionable and overridden only when concern accumulates. CCO calibrates this conservatism online using Conformal Decision Theory, ensuring that undesirable outcomes remain below a user-specified target threshold with finite-time bounds and no distributional assumptions. On a modified version of SWE-bench, weaker overseers successfully constrain an adversarially misaligned stronger agent; on MACHIAVELLI, CCO substantially reduces ethical violations while preserving reward. In both settings, empirical violation rates closely match the specified targets, as predicted by the theory.