Search papers, labs, and topics across Lattice.
This paper investigates how language models respond to instructions to underperform on multiple-choice evaluations, finding that instruction complexity dictates whether models engage with question content or rely on positional shortcuts. By testing Llama-3 models on MMLU-Pro with varying adversarial instruction specificity, the study identifies three distinct regimes: vague instructions lead to moderate accuracy reduction with content engagement, standard sandbagging induces positional entropy collapse with partial engagement, and a complex, two-step instruction causes extreme positional collapse and content insensitivity. The attractor position in the collapse matches the model's default behavior, highlighting the influence of instruction complexity on adversarial compliance mechanisms.
Complex, multi-step instructions can cause LLMs to completely ignore question content and instead rely on positional shortcuts when asked to underperform, revealing a critical vulnerability in adversarial evaluation.
When instructed to underperform on multiple-choice evaluations, do language models engage with question content or fall back on positional shortcuts? We map the boundary between these regimes using a six-condition adversarial instruction-specificity gradient administered to two instruction-tuned LLMs (Llama-3-8B and Llama-3.1-8B) on 2,000 MMLU-Pro items. Distributional screening (response-position entropy) and an independent content-engagement criterion (difficulty-accuracy correlation) jointly characterise each condition. The gradient reveals three regimes rather than a monotonic transition. Vague adversarial instructions produce moderate accuracy reduction with preserved content engagement. Standard sandbagging and capability-imitation instructions produce positional entropy collapse with partial content engagement. A two-step answer-aware avoidance instruction produces extreme positional collapse, with near-total concentration on a single response position (99.9% and 87.4%) and no measurable content sensitivity. This was the only multi-step instruction tested, and it produced the most extreme shortcut. The attractor position matches each model's content-absent null-prompt default. The effect replicates across both models and four academic domains. Distributional collapse and content engagement can co-occur (50% concordance between screening criteria), indicating that entropy-based screening and difficulty-based content assessment capture partially independent dimensions of response validity. Results suggest that instruction complexity can determine whether adversarial compliance uses content-aware or content-blind mechanisms in small instruction-tuned LLMs under greedy decoding.
