Search papers, labs, and topics across Lattice.
This paper introduces adversarial pragmatics as a comprehensive benchmark and annotation protocol designed to evaluate language model behavior in complex scenarios involving instruction conflict, embedded commands, and other linguistic ambiguities. By establishing a linguistically controlled taxonomy and a robust evaluation framework, the authors provide a means to differentiate between various failure modes, such as capability limits and policy ambiguities, which are often conflated in existing benchmarks. The key result is a validated 18-item seed benchmark that enhances the precision of safety evaluations and improves the understanding of model behavior under nuanced linguistic conditions.
Adversarial pragmatics reveals that existing safety evaluations often obscure critical distinctions in model behavior, making it harder to pinpoint the root causes of failures.
Safety evaluations for language models increasingly depend on judgments about ambiguous natural-language behaviour: whether a model has followed an instruction, refused appropriately, complied with a policy, resisted an embedded command, or misreported progress in an agentic task. Existing benchmarks often compress these distinctions into pass/fail labels, obscuring whether failures arise from capability limits, policy ambiguity, instruction conflict, scaffold failure, or unstable evaluator judgments. This paper introduces adversarial pragmatics as a benchmark and annotation protocol for evaluating model behaviour under instruction conflict, embedded commands, quotation, scope ambiguity, deixis, indirect speech acts, and multi-turn agent transcripts. The contribution is empirical and methodological: a linguistically controlled taxonomy, an 18-item seed benchmark with validator-enforced metadata, a 54-row local seed pilot, an expert-evaluation protocol distinguishing task success, policy compliance, safety risk, refusal outcome, and evaluator confidence, and metrics for judge validity, diagnostic ambiguity, and taxonomy drift. The framework turns linguistic judgment methodology into a practical tool for validating safety evals, LLM judges, gold-set construction, prompt-injection tests, and safety documentation.
