Search papers, labs, and topics across Lattice.
This study evaluates the performance of open-weight large language models (LLMs) in automating data preparation tasks essential for longitudinal population studies, addressing the limitations posed by governance restrictions on data transmission to cloud services. By developing a comprehensive framework that includes a curated dataset and specific task definitions, the authors benchmark the efficacy of various LLMs in generating R code for data cleaning and merging tasks. The results indicate that consumer-grade open-weight models can achieve up to 87.9% average task completion, suggesting a promising alternative for AI-assisted data preparation in sensitive research contexts.
Open-weight LLMs can achieve nearly 88% task completion in complex data preparation tasks without ever sending sensitive data to the cloud.
Large language models (LLMs) and agents are now widely used tools in code development, with data typically sent to third-party cloud-based models. Their adoption in research using personal data is constrained by governance requirements that typically prohibit data transmission to external services. Locally deployable open-weight models offer an alternative since sensitive data never leave the local environment. We introduce an open-source framework for evaluating the efficacy of AI agents powered by open-weight LLMs on one of the most persistent bottlenecks in research on longitudinal population studies: data preparation. The framework comprises: a curated ground-truth dataset (cleaning scripts preparing six sweeps of data from a British cohort study), task definitions encompassing tasks such as category harmonization and multi-wave merging, and automated routines for evaluating the LLM-produced R code and outputted data. We benchmark LLMs across the (consumer grade) deployment spectrum to assess their efficacy in 20 data preparation tasks (creation of 102 variables). Current state-of-the-art, 31-35B parameter models almost saturated our benchmark ("average task completion" up to 87.9%). The performance of open-weight LLMs running on consumer-grade hardware shows promise of a viable path toward AI-assisted data preparation in governance-restricted research settings. Our framework is publicly available at: https://github.com/UCL-ARC/RRBench.
