Search papers, labs, and topics across Lattice.
This paper benchmarks LLMs as text classifiers on 15 biomedical datasets, systematically varying prompt types, output processing, few-shot examples, and model scale. They find zero-shot LLMs achieve a PR AUC of 0.4, while few-shot prompting reaches nearly 0.5, approaching the performance of naive Bayes and random forest classifiers. The study highlights the effectiveness of using output token probabilities for class probability prediction in this domain.
LLMs can classify biomedical articles surprisingly well, rivaling traditional methods like Naive Bayes and Random Forests, especially when using output token probabilities.
This work presents a systematic and in-depth investigation of the utility of large language models as text classifiers for biomedical article classification. The study uses several small and mid-size open source models, as well as selected closed source ones, and is more comprehensive than most prior work with respect to the scope of evaluated configurations: different types of prompts, output processing methods for generating both class and class probability predictions, as well as few-shot example counts and selection methods. The performance of the most successful configurations is compared to that of conventional classification algorithms. The obtained average PR AUC over 15 challenging datasets above 0.4 for zero-shot prompting and nearly 0.5 for few-shot prompting comes close to that of the na\"ive Bayes classifier (0.5), the random forest algorithm (0.5 with default settings or 0.55 with hyperparameter tuning) and fine-tuned transformer models (0.5). These results confirm the utility of large language models as text classifiers for non-trivial domains and provide practical recommendations of the most promising setups, including in particular using output token probabilities for class probability prediction.
