Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper evaluates the effectiveness of existing AI safety benchmarks designed for large language models (LLMs) when applied to small language models (SLMs). Through a comprehensive assessment of five benchmark suites across 26 open-source SLMs, the authors find that ambiguous judgments are prevalent and correlate with various model characteristics, indicating that LLM-centric benchmarks are inadequate for SLM safety evaluation. The study reveals that ambiguity significantly affects model rankings, highlighting the need for tailored safety assessments for SLMs to avoid misleading conclusions about their safety and performance.
Ambiguity in safety evaluations leads to misleading rankings, revealing that LLM-centric benchmarks are ill-suited for assessing small language models.
Small Language Models (SLMs) are increasingly deployed in resource-constrained, privacy-sensitive settings, where safety and bias failures can cause security and societal risks. However, existing AI safety\slash security\slash compliance benchmarks are designed for large language models that may not transfer reliably to SLMs. We therefore ask: Can these benchmarks effectively and reliably evaluate SLMs? To answer this question, we conduct a large-scale assessment of the effectiveness and robustness of these automated pipelines by evaluating five widely used benchmark suites across 26 open-source SLMs under a unified judging rubric, which assigns a score of 0, 1, or 0.5 to harmful, safe, or ambiguous/irrelevant responses, respectively. Across the benchmarks, ambiguous judgments dominate and correlate with prompt complexity and model architecture, indicating that {\em LLM-centric safety benchmarks are insufficient as standalone evidence for SLM safety assessment}. In general, the ambiguity rate increases with lexical density, output perplexity, and output length and decreases with lexical sophistication, self-coherence, and reply-prompt similarity. This reveals a capability-safety confound that mixes model capability with apparent safety. Since ambiguity is prevalent, aggregate mean-score leaderboards are mathematically brittle: model rankings change significantly under reasonable ambiguity treatments, even when the underlying outputs remain unchanged.