Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study reveals that language model (LM) ratings of depression are significantly more influenced by the LM's specific configuration (model choice and prompting strategy) than by the patient's actual condition, with LM choice explaining 30% of score variance compared to participant differences at 10.5%. Even highly accurate LMs disagreed on screening decisions for 40% of individuals, highlighting substantial rater dependence. While calibration can improve accuracy and reduce disagreement, it does not fully resolve individual-level assessment discrepancies.
Language model depression assessments are overwhelmingly driven by the LM's configuration, not the patient's symptoms, leading to 40% disagreement between even high-performing raters.
Depression has no diagnostic blood test. Language models promise tireless, consistent assessment, but can accurate raters disagree about individuals? We pre-registered 880 language-model raters, crossing 11 open models with prompting and scoring choices, and applied them to 189 interviews against the eight-item Patient Health Questionnaire. Model choice explained 30.0% of summed-symptom score variance, stable participant differences 10.5%. Two randomly drawn raters with area under the receiver operating characteristic curve (AUC)>= 0.70 disagreed on screening decisions for 40% of participants, on average. Average over-rating governed how many were flagged, yet equal-capacity raters chose differently for about one participant in five. A locked analysis of 86 new interviews reproduced the main pre-registered findings. Exploratory recalibration with 40 labelled participants raised accuracy from about 60% to 75% and halved disagreement, leaving one participant in five decided differently. Calibration repaired much of the rater dependence without securing agreement about individuals.