Search papers, labs, and topics across Lattice.
This paper investigates the sensitivity of 23 LLMs to meaning-preserving lexical and syntactic variations in evaluation prompts across MMLU, SQuAD, and AMEGA benchmarks. Using synonym substitution and dependency-parsing-based syntactic transformations, the authors demonstrate that lexical perturbations consistently degrade performance, while syntactic perturbations have variable effects. The study reveals that LLM performance is highly sensitive to surface-level lexical variations, challenging the reliability of current benchmarks and highlighting the need for robustness testing.
LLM benchmark scores can swing wildly with simple synonym swaps, revealing a surprising reliance on superficial lexical cues rather than deeper understanding.
The rapid advancement of Large Language Models (LLMs) has established standardized evaluation benchmarks as the primary instrument for model comparison. Yet, their reliability is increasingly questioned due to sensitivity to shallow variations in input prompts. This paper examines how controlled, truth-conditionally equivalent lexical and syntactic perturbations affect the absolute performance and relative ranking of 23 contemporary LLMs across three benchmarks: MMLU, SQuAD, and AMEGA. We employ two linguistically principled pipelines to generate meaning-preserving variations: one performing synonym substitution for lexical changes, and another using dependency parsing to determine applicable syntactic transformations. Results show that lexical perturbations consistently induce substantial, statistically significant performance degradation across nearly all models and tasks, while syntactic perturbations have more heterogeneous effects, occasionally improving results. Both perturbation types destabilize model leaderboards on complex tasks. Furthermore, model robustness did not consistently scale with model size, revealing strong task dependence. Overall, the findings suggest that LLMs rely more on surface-level lexical patterns than on abstract linguistic competence, underscoring the need for robustness testing as a standard component of LLM evaluation.
