Search papers, labs, and topics across Lattice.
This paper introduces a novel framework called debiased inference with multiple imperfect measurements (DMM) that addresses the biases introduced by using AI-generated data without gold-standard labels. By leveraging multiple error-prone measurements and assuming conditional independence, DMM enables valid downstream inference while maintaining statistical rigor, even in the presence of unknown misclassification rates. Simulation results demonstrate that DMM not only yields valid inference but also enhances efficiency when incorporating accurate yet imperfect measurements, making it particularly relevant for social science applications involving large language model annotations.
Valid inference from AI-generated data is possible without gold-standard labels, challenging the conventional reliance on costly benchmarks.
An increasing number of scholars use AI to measure variables they subsequently include in downstream analyses. Although AI-measured variables are often analyzed as if observed without error, ignoring prediction errors in automated measurement leads to substantial bias and invalid confidence intervals in downstream analyses, even if AI measurement accuracy is high, e.g., above 90%. Existing solutions, such as design-based supervised learning and prediction-powered inference, combine error-prone AI-based measurements with gold-standard labels, which may be costly and difficult to obtain in some application areas. In this paper, we propose debiased inference with multiple imperfect measurements (DMM), a framework that combines multiple error-prone AI measurements to enable valid downstream inference without gold-standard labels. Building on the established results on CP decomposition, DMM assumes that these measurements are independent conditional on the latent true label and observed unit-level features, such as text features represented by embeddings. This framework allows for unknown misclassification rates to vary across annotation methods (e.g., large language models) and across units of annotation (e.g., texts). Under this assumption, we use semiparametric inference theory to prove that the DMM estimator is consistent and asymptotically normal, enabling valid inference for a wide range of downstream statistical analyses common in the social sciences. Our simulation results show that DMM yields valid inference and that adding accurate, though imperfect, measurements can improve efficiency. Focusing on common applications of large language model annotations, we also develop diagnostics to assess the conditional independence assumption.
