Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the challenges of fairness auditing in AI systems where auditors have limited access to model outputs, proposing a sequential hypothesis-testing framework that adapts to real-world constraints. By formulating fairness auditing as a tolerance-aware problem, the authors demonstrate that both the choice of fairness metric and the level of model access significantly impact the efficiency of audits. Key findings reveal that richer model outputs can reduce the number of queries needed for certain fairness metrics, although the benefits vary depending on the auditing context.
Richer model outputs can drastically cut down the number of queries needed for fairness audits, but the gains are context-dependent.
External evaluations are becoming increasingly central to the governance of AI systems. In practice, however, independent auditors often have limited access to deployed models and must rely on query-based interactions. Most existing fairness evaluation methods assume static datasets and fixed-sample statistical tests, making them poorly suited to real-world auditing scenarios in which evidence must be collected sequentially under query constraints. In this work, we formulate fairness auditing as a tolerance-aware sequential hypothesis-testing problem under limited model output access. We develop a sequential generalized likelihood-ratio framework that allows auditors to accumulate evidence from a finite audit pool and stop once sufficient support for compliance or violation has been obtained. The framework is instantiated for decision-based Statistical Parity and Equal Opportunity audits, and extended to score- and logit-based proxy audits when richer observables are available. Our results show that both the fairness metric and the level of model access significantly affect audit efficiency, and that the benefits of richer output information are not uniform across auditing settings. In particular, richer outputs can substantially reduce the number of queries required for some fairness metrics and operating regimes, while offering limited gains in near-threshold cases. This work provides a practical statistical framework for sequential fairness auditing under realistic deployment constraints.