Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To isolate post-tool-failure reporting dishonesty from execution and environment dynamics, the authors introduce Failure-Transparent Agents (FTA), a benchmark of 100 deterministic failure traces across five failure families and four user-pressure conditions. Evaluating whether agents falsely claim completion or hallucinate details is critical because silent reporting failures compound underlying tool errors in autonomous systems. Across six models and 3,600 human-annotated responses, enforcing a structured evidence contract reduced false-success rates from 22.8% to 0.8% and fabricated details from 28.3% to 0.8%, while concurrently improving useful recovery responses from 74.9% to 98.8%.
Tool-using agents falsely claim success in nearly a quarter of tool failure cases, but enforcing structured evidence contracts slashes these deceptive reporting errors to under 1% while maximizing task recovery.
Tool-using agents can fail twice: a required tool can fail, and the agent can then report success without the evidence needed to justify it. Existing benchmarks often entangle this reporting failure with tool selection, recovery, and environment dynamics. We introduce Failure-Transparent Agents (FTA), a controlled benchmark that fixes the failed observation and required evidence state before generation, making post-failure claims directly auditable. FTA contains 100 tasks with deterministic failure traces spanning five failure families, a neutral control, and four user-pressure conditions, and evaluates unsupported claims alongside useful recovery. Across six models, three response policies, and 3,600 human-annotated responses, false-success rates are 22.8% under the baseline policy, 9.3% with a transparency instruction, and 0.8% with a structured evidence contract. Fabricated-detail rates decrease from 28.3% to 14.3% and 0.8%, while useful responses increase from 74.9% to 89.2% and 98.8%, respectively. The tested evidence-contract policy is associated with substantially lower post-failure reporting errors while useful-response rates remain high within this blocked-task benchmark.
Tsinghua AI