Search papers, labs, and topics across Lattice.
This paper analyzes physician disagreement in the HealthBench medical AI evaluation dataset, identifying factors contributing to variance in met/not-met labels. The study finds that rubric identity accounts for a small portion of disagreement variance (3.6-6.9%), while the majority of variance (81.8%) remains unexplained by metadata, rubric language, specialty, or embeddings. The analysis reveals that disagreement is highest for borderline cases and significantly increases with reducible uncertainty (missing context), suggesting that improving information clarity in evaluation scenarios could reduce disagreement.
Physician disagreement in medical AI evaluation isn't about who's judging or what the rubric says, but mostly boils down to case-specific factors and fixable uncertainties in the evaluation setup.
We decompose physician disagreement in the HealthBench medical AI evaluation dataset to understand where variance resides and what observable features can explain it. Rubric identity accounts for 15.8% of met/not-met label variance but only 3.6-6.9% of disagreement variance; physician identity accounts for just 2.4%. The dominant 81.8% case-level residual is not reduced by HealthBench's metadata labels (z = -0.22, p = 0.83), normative rubric language (pseudo R^2 = 1.2%), medical specialty (0/300 Tukey pairs significant), surface-feature triage (AUC = 0.58), or embeddings (AUC = 0.485). Disagreement follows an inverted-U with completion quality (AUC = 0.689), confirming physicians agree on clearly good or bad outputs but split on borderline cases. Physician-validated uncertainty categories reveal that reducible uncertainty (missing context, ambiguous phrasing) more than doubles disagreement odds (OR = 2.55, p<10^(-24)), while irreducible uncertainty (genuine medical ambiguity) has no effect (OR = 1.01, p = 0.90), though even the former explains only ~3% of total variance. The agreement ceiling in medical AI evaluation is thus largely structural, but the reducible/irreducible dissociation suggests that closing information gaps in evaluation scenarios could lower disagreement where inherent clinical ambiguity does not, pointing toward actionable evaluation design improvements.
