Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the limitations of human-in-the-loop oversight for LLM agents executing irreversible actions, revealing that human reviewers exhibit only moderate agreement on what constitutes a "risky" action. By framing the oversight mechanism as a selective classification problem under asymmetric costs, the authors demonstrate that increased human oversight can paradoxically reduce safety due to reviewer fatigue, leading to an inverted-U relationship between oversight and safety. The study introduces an open-source agent-oversight system that quantifies these dynamics, transforming the evaluation of oversight effectiveness from subjective judgment into measurable metrics.
Increased human oversight can paradoxically make LLM agents less safe due to reviewer fatigue, challenging conventional safety assumptions.
As LLM agents begin to take real, irreversible actions (shell commands, file edits, deploys), the standard safety pattern is a human-in-the-loop approval gate: risky actions pause and wait for a person. We argue the gate is the easy part; the hard part is the judgment - which actions to stop - which the field evaluates against two false assumptions: that there is a ground-truth notion of"risky,"and that the human reviewer is a perfect, infinitely-available oracle. On a hand-labeled set of 125 adversarially-weighted agent actions we show that (i) reviewers only moderately agree on what is risky (Fleiss'kappa = 0.52), so there is no single correct label; (ii) framing the guard as selective classification under asymmetric cost makes its operating limits measurable, and on hard inputs the guard cannot safely auto-decide; and (iii) when the reviewer is modeled as endogenous (fatiguing as escalation load grows), realized safety becomes an inverted-U in the escalation rate: more human oversight can make a system less safe, and the safety-optimal guard escalates below full escalation - a setting a load-aware policy also uses to resist a flooding attack that slips a malicious action past a fatigued reviewer. Agent oversight, framed this way, is not only a classification problem but a resource-allocation one: human attention is finite, and the guard's escalation policy spends it. We claim none of these mechanisms as novel - fatigue-aware learning-to-defer (FALCON), cost-sensitive deferral under workload constraints (DeCCaF), trajectory-level guarding, and reviewer-fatigue/flooding attacks are all prior art we cite. Our contribution is an open-source agent-oversight system that operationalizes and measures them in the LLM-agent action-gating setting, turning"is my guard good?"from a guess into a curve. The inverted-U and the flooding attack are modeling results that motivate a human study.