Search papers, labs, and topics across Lattice.
This study conducts a systematic zero-shot evaluation of 41 open-weight language models across various parameter sizes and datasets to provide guidance for selecting models for intent classification in task-oriented dialogue systems. The evaluation reveals that instruction-tuned 3B models can outperform some 7B models, while also highlighting the saturation of widely used benchmarks like SNIPS, which fail to differentiate among current models. Additionally, the research assesses multiple factors such as confidence calibration and robustness, offering a comprehensive framework for model selection under practical constraints.
Instruction-tuned 3B models can outperform larger 7B models in intent classification, challenging assumptions about model size superiority.
Intent classification is a core component of task-oriented dialogue systems, yet practitioners have limited systematic guidance for selecting deployable open-weight language models under compute, latency, and robustness constraints. We present a systematic zero-shot evaluation of 41 open-weight language models spanning 15 families and the 135M--9B parameter range across eight English single-label intent-classification datasets. A ninth dataset, ATIS, uses five labeled demonstrations and is reported as an auxiliary five-shot result. The evaluation includes standard benchmarks, a large-scale voice-assistant corpus, and production-derived e-commerce datasets. Beyond exact-match accuracy, we analyze confidence calibration, robustness to realistic input perturbations, statistical reliability of model rankings, deployment efficiency, and benchmark saturation. Our results show that instruction-tuned 3B models can outperform several evaluated 7B base models, that differences among leading models on MASSIVE are statistically indistinguishable under pairwise McNemar tests, and that widely used benchmarks such as SNIPS have become saturated and no longer meaningfully discriminate among current open-weight models. Instruction tuning's effect on confidence calibration is inconsistent rather than uniformly harmful. These findings provide practical guidance for selecting and evaluating open-weight language models for intent classification.
