Search papers, labs, and topics across Lattice.
This paper investigates fine-tuning a quantized 1.7B parameter Small Language Model (SLM) on limited human-annotated data for text evaluation and annotation. They introduce a custom, multi-dimensional rubric framework along with data augmentation and regularization techniques to improve alignment with human expert consensus. The resulting SLM achieves a 0.23 increase in Krippendorff's $\alpha$ compared to state-of-the-art proprietary LLMs, demonstrating the potential of task-specific alignment and efficient quantization.
Forget giant models: A carefully trained, quantized SLM can beat proprietary LLMs at aligning with human annotators.
As Large Language Model (LLM) capabilities advance, the demand for high-quality annotation of exponentially increasing text corpora has outpaced human capacity, leading to the widespread adoption of LLMs in automatic evaluation and annotation. However, proprietary LLMs often exhibit systematic biases that diverge from human expert consensus, lacks reproducibility, and raises data privacy concerns. Our work examines the viability of finetuning a quantized Small Language Model of 1.7B parameter size on limited human-annotated data to serve as a highly aligned, deterministic evaluator and annotator. By implementing a custom, multi-dimensional rubric framework and simple augmentation and regularization techniques, the proposed approach achieves higher inter-annotator agreement (0.23 points increase in Krippendorff's $\alpha$) than the best performing state-of-the-art proprietary LLM. We also demonstrate the generalizability of the proposed training pipeline on a separate emotion classification task. The results show that task-specific alignment and efficient 4-bit quantized fine-tuning provide superior open-source alternative to using proprietary models for evaluation and annotation. Our finetuning approach is publicly available at https://github.com/jylee-k/slm-judge.
