Track genomics, neuroscience and molecular biology.
Search papers, labs, and topics across Lattice.
This paper introduces psychometric representational similarity analysis (PRSA) to disentangle questionnaire wording effects from genuine person-level psychological signal in language model assessments of psychiatric symptoms. By comparing the geometric structure of speech-derived scores, self-report, item wording, and theory, the authors demonstrate that language-derived symptom geometry often reflects item wording covariance rather than psychological structure. The COMPASS 2.0 framework, incorporating PRSA and person-level construct scoring, reveals that while language models can capture self-report agreement, this agreement can be artifactual, highlighting the need for methods that distinguish between shared wording and individual psychological states.
Language models scoring psychiatric questionnaires often mimic item wording, not individual psychological states, a critical distinction revealed by a novel representational similarity analysis.
Language models can score psychiatric questionnaires from speech, but agreement with self-report may reflect the questionnaire rather than the person. We introduce psychometric representational similarity analysis, a framework for comparing the structure of speech-derived scores, self-report, item wording and theory, and implement it alongside person-level construct scoring in COMPASS 2.0. We show how similarly worded items induce covariance without psychological signal. In pre-registered discovery and confirmation analyses of clinical interviews from 275 participants, language-derived symptom geometry resembled wording more than self-report, with no structure beyond wording detected by the registered tests. Geometric agreement with self-report survived assigning participants someone else's answers, whereas person-paired scores captured distress more than specific symptoms. Complementary analyses examined counselling quality and wording structure across 34 instruments and the Research Domain Criteria (RDoC) framework. These findings distinguish agreement about psychological structure from evidence that language-derived assessments track individual people.