Search papers, labs, and topics across Lattice.
This paper introduces Adaptive Matrix Validation (AMV), a novel approach for enhancing the accuracy of AI-assisted interviews by integrating structured validation questions to adjust mapped survey data. The method addresses the inherent fallibility of AI mapping by calibrating responses using both peer validation and individual respondent data, leading to improved estimates for item means, subgroup analyses, and regression coefficients. Through simulations and real-world emulations, the authors demonstrate that sparse validation can significantly enhance precision in certain contexts, providing a framework for optimizing survey methodologies in AI applications.
Sparse validation in AI-assisted interviews can dramatically improve data accuracy, revealing when and how to leverage structured questions effectively.
AI-assisted interviews promise to reduce respondent burden in surveys by allowing respondents to describe experiences naturally while an AI system noisily maps those accounts into structured survey variables. That mapping is a measurement process that is fallible, versioned, adaptive, and potentially behaves differently across subgroups. This paper proposes Adaptive Matrix Validation (AMV), a design in which each respondent completes an AI-assisted interview, which is then mapped into tabular data by the AI. Respondents are also asked a small, randomized set of structured questions, which are used for statistical adjustment. The estimator first calibrates the mapped values using validation answers from other respondents, then corrects the remaining error with the validation answers observed for the target respondent. The paper develops estimators for item means, subgroup estimates, and regression coefficients when outcomes, predictors, or both are mapped from interviews. It also gives planning formulas the number of validation questions required and the sample size. A design-calibration simulation, an American Time Use Survey emulation, and a CHAMPS verbal-autopsy narrative study show when sparse validation can improve precision and when it cannot
UW