Search papers, labs, and topics across Lattice.
This study investigates the behavior of small language models when faced with conflicting instructions across three tasks: multiple-choice question answering, sentiment classification, and mathematical question answering. By pairing standard instructions with non-standard ones, the authors reveal that while model accuracy improves with size, small models frequently ignore conflicting instructions, indicating a disconnect between task competence and instruction adherence. The findings highlight that relying solely on standard accuracy can obscure significant instruction-following failures, suggesting that these two capabilities should be evaluated separately.
Small language models may excel at tasks but often disregard conflicting instructions, revealing a critical gap in their reliability.
Instruction tuning is meant to make language models follow user requests, yet it is unclear whether small models comply when an instruction conflicts with their usual task behavior. We study this across three tasks - multiple-choice question answering (MCQA), sentiment classification, and mathematical question answering - by pairing a standard instruction with a conflicting non-standard one (select an incorrect option, output the opposite sentiment, or return twice the answer). This cross-task design allows us to test whether resistance to conflicting instructions is tied to specific task characteristics or reflects a broader behavioral tendency. As all predictions are scored against the original ground truth, a model that ignores the non-standard instruction still appears accurate. Using standard accuracy, non-standard accuracy, and an Instruction-Following Failure Rate (IFFR), we evaluate instruction-tuned Qwen models across sizes. Both standard accuracy and instruction following generally improve with scale, although the pattern is not consistent across all tasks and datasets. Small models stay competent yet routinely ignore the non-standard instruction, while larger models show a clear gap between the two settings. These findings suggest that gains in task capability do not automatically provide reliable control over model behavior. Task competence and instruction following are therefore distinct abilities, and reporting only standard accuracy hides instruction-following failures.
