Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces TQLite, a distillation framework that enables small language models (SLMs) to achieve near the translation quality evaluation performance of large reasoning models (LRMs) by utilizing a multi-LRM jury for generating high-quality synthetic training data. The study benchmarks SLMs, LLMs, and LRMs across various translation quality evaluation setups, revealing that SLMs trained with TQLite significantly outperform standard SLMs in translation quality metrics. This advancement is crucial as it offers a scalable and cost-effective solution for real-time translation quality evaluation, addressing the computational challenges associated with deploying LLMs and LRMs at scale.
SLMs can now rival the translation quality evaluation performance of LRM-based models, thanks to a novel distillation framework that leverages a multi-LRM jury for data generation.
Large language models (LLMs) have demonstrated impressive performance in MQM-based translation quality (TQ) evaluation, and recent advances in large reasoning models (LRMs) promise even greater improvements. However, both LLMs and LRMs are computationally expensive to deploy at scale, while small language models (SLMs)---though much more efficient---struggle with the complex reasoning required for evaluation tasks. In this work, we present an extensive empirical study benchmarking SLMs, LLMs, and LRMs across a wide range of TQ evaluation setups, providing a comprehensive view of the current landscape and establishing best practices. To address the scalability challenge, we introduce TQLite, a novel distillation framework that enables SLMs to approach the MQM evaluation performance of the best LRM-based evaluators. Our approach leverages a multi-LRM jury to generate high-quality synthetic training data via practical data curation techniques and aggregation of evaluation responses across a diverse panel of models. Our results demonstrate that SLMs trained via TQLite achieve strong MQM evaluation performance that far exceeds off-the-shelf evaluation capabilities of standard SLMs, offering a scalable and cost-effective alternative to LLM- and LRM-based evaluators.