Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces VFR-Audit, a novel framework that quantifies the stability of fairness audit verdicts in hospital length-of-stay predictions by calculating the Verdict Flip Rate (VFR), which indicates the likelihood of verdict changes across audits. The framework addresses the limitations of existing uncertainty methods that fail to translate continuous fairness metrics into reliable binary verdicts, particularly in the context of bias-mitigation strategies. Key findings reveal that VFR-Audit not only enhances verdict reliability but also provides insights into the effects of model discrimination on fairness outcomes across different hospital settings.
Verdict stability in fairness audits can swing dramatically, but VFR-Audit quantifies this uncertainty, revealing how bias-mitigation strategies impact reliability.
Fairness audits in clinical Artificial Intelligence convert continuous fairness metrics into binary pass-or-fail verdicts against operational thresholds, where hospital governance boards, payers, and regulators act on the resulting verdicts. Such audits are repeated over time and across hospital sites, thus the same verdict can flip between pass and fail across audits. Existing uncertainty methods such as Bayesian posteriors, bootstrap confidence intervals, and permutation tests address verdict instability only at the continuous-metric level. Converting metric-level uncertainty into a verdict-stability claim remains a manual step that scales poorly across the (model, metric, attribute) cells an audit covers. Existing uncertainty methods also leave open whether bias-mitigation steps, such as reweighing or per-group threshold shifts, yield a stable passing verdict at the cost of model discrimination measured as AUROC or AUPRC.To address this verdict-stability gap, we propose VFR-Audit, a framework built around the Verdict Flip Rate (VFR), a scalar bounded between 0 and 0.5 that measures the probability of verdict reversal under stratified bootstrap resampling. VFR-Audit reports VFR alongside three reliability axes, namely within-cohort resampling stability, audit-size sensitivity, and cross-hospital verdict agreement via Fleiss' kappa.