Search papers, labs, and topics across Lattice.
This study fine-tunes three small LLMs (Phi-3-mini, Qwen2.5-3B, and Mistral-7B) using QLoRA on biomedical datasets SciFact and HealthVer, achieving significant performance improvements over larger models like GPT-4o and GPT-5. Mistral-7B, in particular, outperforms these larger models by up to 12% in F1 score while utilizing only 1,008 training examples, demonstrating the effectiveness of small models in this domain. Additionally, the research uncovers a structural artifact in SciFact that inflates in-domain scores, emphasizing the importance of dataset integrity for reliable cross-domain generalization.
Small LLMs can outperform larger models in biomedical claim verification, achieving significant gains at a fraction of the cost.
Large Language Models such as GPT-4o and GPT-5 achieve strong zero-shot performance on biomedical claim verification, but cost and opacity limit scalable use. We fine-tune three small LLMs: Phi-3-mini (3.8B), Qwen2.5-3B, and Mistral-7B, via QLoRA on SciFact and HealthVer, providing the first study of QLoRA models against GPT-4o and fine-tuned BioLinkBERT encoders. Mistral-7B QLoRA surpasses both GPT-4o and GPT-5 (up to 12% F1 gain) at a fractional cost using just 1,008 training examples. We conduct extensive in-domain and cross-domain evaluation: models trained on SciFact tested on HealthVer and vice versa, at matched sizes to isolate dataset structure from data quantity. We identify a previously unreported structural artifact in SciFact that inflates in-domain scores, and show through bidirectional out-of-domain evaluation that training on structurally sound data enables robust cross-domain transfer. We plan to release all code and adapter checkpoints.
