Search papers, labs, and topics across Lattice.
This paper introduces Cautious Bench, a novel benchmark designed to evaluate the over-safety of agent guardrails in large language models (LLMs) by mapping the decision boundary of authorized versus unauthorized actions. The authors demonstrate that existing guardrails often refuse legitimate actions based on the perceived threat level of object names, revealing a significant name-superstition effect where guardrails are more likely to block safe actions associated with "scary" labels. This finding highlights a critical flaw in current safety mechanisms, as it shows that guardrails prioritize superficial labels over contextual authorization, which can hinder the deployment of LLMs in practical applications.
Guardrails are more likely to block safe actions when faced with "scary" object names, exposing a critical flaw in LLM safety mechanisms.
Agent guardrails are checks that approve or refuse each action before an LLM executes it. Sometimes they refuse requests that are genuinely safe. This over-safety blocks deployment when a guardrail refuses an authorized task. Evaluating over-safety is hard: at the boundary an authorized action resembles an unauthorized one, and the safe-versus-unsafe label is a choice of authorization policy, not fixed by the action alone. We argue it therefore requires a benchmark that does not yet exist, one that maps the decision boundary of an ideal guardrail. Harvesting such a benchmark from real data is impractical: boundary cases are hard to collect, their labels hard to verify. The gap is real, so we construct Cautious Bench, the first benchmark to make over-safety the construct for agent guardrails; it codesigns each sample and its label with a stated authorization policy. A build-time gate re-derives every example to certify it, so each label is a mechanical consequence of the policy rather than an annotator's per-sample verdict, a reference against which researchers can measure real guardrails. The benchmark renders 756 Decidable benign/twin pairs, each under three object-name types (2,268 measured pairs), and 40 Undecidable pairs reported separately. Measuring six guardrails from five designs, we find a name-superstition effect: each over-refuses an authorized action more often under a scary-looking object name than a benign one. Since only the object name varies in the aforementioned contrast experiments, the deviation is the name's doing: the guardrails read the surface label, not the authorization context.
