Search papers, labs, and topics across Lattice.
This study evaluates the performance of small language models (SLMs) ranging from 360M to 3B parameters in relation extraction (RE) tasks, comparing them against zero-shot frontier LLMs and a discriminative RoBERTa baseline. The findings reveal that the best-performing SLM, Qwen2.5-0.5B, achieves a micro-F1 score of 0.83 in general-domain tasks, outperforming GPT-5.4 and Claude Sonnet 4.6, highlighting the effectiveness of targeted task adaptation over general-purpose models. Additionally, tuned SLMs excel in literary RE, achieving a score of 0.92 on the Biographical benchmark, demonstrating that compact, task-adapted models can deliver high accuracy while being resource-efficient and privacy-preserving.
Small language models can outperform leading zero-shot LLMs in relation extraction tasks when fine-tuned on task-specific data, challenging the notion that bigger is always better.
Large language models (LLMs) achieve strong relation extraction (RE), but their computational demands and reliance on proprietary APIs limit deployment in resource-constrained or privacy-sensitive settings. We investigate how far small language models (SLMs) can close this gap across general-domain and literary text. We evaluate five models from 360M to 3B parameters under three domain-composition regimes and two prompt-conditioned tuning styles (30 configurations), comparing them with zero-shot frontier LLMs and a discriminative RoBERTa baseline. Across nine benchmarks, the best sub-billion model, Qwen2.5-0.5B fine-tuned on pooled general-domain data, achieves a general-domain positive-class micro-F1 of 0.83, versus 0.69 for GPT-5.4 and 0.66 for Claude Sonnet 4.6 evaluated zero-shot. This does not imply that SLMs are intrinsically stronger; rather, targeted task adaptation enables 4-bit models deployable on a single consumer GPU to outperform general-purpose frontier systems under this protocol. An in-domain RoBERTa baseline also exceeds both frontier models, indicating that the gain stems from task adaptation rather than generative decoding. On literary RE, tuned SLMs reach 0.92 on the human-annotated Biographical benchmark versus 0.83 for GPT-5.4, and 0.833 versus 0.578 on the two-benchmark literary average. A targeted domain-adaptive pretraining case study yields no practically meaningful gain over supervised fine-tuning, while the cleanest within-family scale comparison shows only marginal improvement. These results show that, when task-specific data are available, compact task-adapted models can provide accurate, private, and hardware-efficient RE.
