Search papers, labs, and topics across Lattice.
This paper investigates the reliability of LLM-generated audit reports by examining how different evidence interfaces influence the reports' validity and reasoning. Using a controlled evaluation framework across 60 AGORA policy cases, the authors generated 600 structured reports under various evidence conditions and assessed them with human reviewers. The findings reveal that while internal evidence alters citation practices, it does not inherently enhance report validity, highlighting a governance risk where reports may appear credible despite referencing irrelevant internal evidence.
Internal evidence can mislead auditors, making reports sound plausible while lacking actual relevance, exposing a critical governance risk in AI oversight.
As AI governance moves from benchmark scores toward auditable oversight, a central question is how reviewers can tell whether an LLM-generated audit report is actually supported by evidence. This paper studies that question in passage-anchored policy audits, where a report must interpret a given policy passage and cite evidence for its claims. We introduce a controlled evaluation framework that holds the passage, rubric, and auditor model fixed while changing only the evidence interface supplied to the auditor. Across 60 AGORA policy cases, we generate 600 structured reports under ten evidence conditions, including passage-based evidence, internal model evidence, a hybrid package, and a shuffled control that preserves evidence format while breaking case relevance. Five human reviewers evaluate the primary interfaces for correctness, passage grounding, diagnostic usefulness, and evidence misuse. The results show that internal evidence changes how reports cite and reason about evidence, but more internal citations do not by themselves make a report more valid. A white-box diagnostic explains the failure mode: causal localization is narrow, while reports readily reuse broader readable labels and token directions. The hybrid interface is the most useful on average, while the shuffled control exposes a key governance risk: reports can sound substantively plausible while citing irrelevant internal evidence. This study reframes internal model access as an evidence design problem for audit workflows, rather than as a guarantee of transparency.
