Search papers, labs, and topics across Lattice.
This paper investigates the use of Llama 3, an open-source large language model (LLM), to analyze trends in clinical trials with decentralized elements (DCTs) using data from ClinicalTrials.gov. The study compares the performance of three Llama 3 models (8b, fine-tuned 8b, and 70b) in classifying DCTs and extracting decentralized elements through prompt engineering. The results show that fine-tuning the 8b model improves sensitivity in DCT classification, while the 70b model is necessary for accurate extraction of decentralized elements, ultimately enabling the identification of 692 DCTs within a 6-year dataset.
Open-source LLMs like Llama 3 can unlock hidden insights in unstructured clinical trial data, but you'll need to carefully balance model size and fine-tuning to get the job done.
Despite interest in clinical trials with decentralized elements (DCTs), analysis of their trends in trial registries is lacking due to heterogeneous designs and unstandardized terms. We explored Llama 3, an open‐source large language model, to efficiently evaluate these trends. Trial data were sourced from Aggregate Analysis of ClinicalTrials.gov, focusing on drug trials conducted between 2018 and 2023. We utilized three Llama 3 models with a different number of parameters: 8b (model 1), fine‐tuned 8b (model 2) with curated data, and 70b (model 3). Prompt engineering enabled sophisticated tasks such as classification of DCTs with explanations and extracting decentralized elements. Model performance, evaluated on a 3‐month exploratory test dataset, demonstrated that sensitivity could be improved after fine‐tuning from 0.0357 to 0.5385. Low positive predictive value in the fine‐tuned model 2 could be improved by focusing on trials with DCT‐associated expressions from 0.5385 to 0.9167. However, the extraction of decentralized elements was only properly performed by model 3, which had a larger number of parameters. Based on the results, we screened the entire 6‐year dataset after applying DCT‐associated expressions. After the subsequent application of models 2 and 3, we identified 692 DCTs. We found that a total of 213 trials were classified as phase 2, followed by 162 phase 4 trials, 112 phase 3 trials, and 92 phase 1 trials. In conclusion, our study demonstrated the potential of large language models for analyzing clinical trial information not structured in a machine‐readable format. Managing potential biases during model application is crucial.
