Search papers, labs, and topics across Lattice.
This study analyzes the impact of prompt engineering on drug toxicity prediction using large language models (LLMs), revealing that minor prompt variations can lead to significant output discrepancies. The research highlights that while fine-tuning prompts can improve results, the inherent variability of LLMs often overshadows these adjustments. Notably, the use of chemoinformatic code to extract features resulted in substantial performance gains over LLM-generated values, indicating a potential direction for enhancing predictive accuracy in drug development.
Minor changes in prompt phrasing can drastically alter LLM outputs, overshadowing the benefits of fine-tuning in drug toxicity prediction.
Clinical trials in the UK can cost up to {\pounds}1.3 million, with approximately 90% drug failure rate. Toxicity is a major contributing factor in drug failure. Testing is time and cost intensive. In recent years, the use of artificial intelligence has been increasingly explored to aid in the prediction of drug toxicity, with extensive use of large language models (LLMs). However, LLMs can show considerable variation when minor changes are made to prompts, which raises concerns about their sensitivity to prompt engineering. Prompt engineering is used to optimise a prompt given to an LLM to generate the desired output. This paper proposes a method to analyse prompt engineering for drug toxicity prediction. The aim of the paper is to investigate the importance of prompt phrasing for drug toxicity prediction. LLMs were prompted to identify chemical properties of significance when predicting drug toxicity. Prompts were constructed to investigate; job role, prompt structuring, and rule interpretation. LLMs were then used to generate datasets, using the identified features from initial prompting, which were then passed to machine learning algorithms. The experiments show that the natural variance which occurs in LLMs outweighs any fine-tuning of prompts. There were, however, substantial improvements in model performance when using chemoinformatic code to extract features instead of using LLM-generated values. The proposed analysis methodology is applicable to a wide range of prompt types across different areas of bioinformatics.
