Search papers, labs, and topics across Lattice.
This paper introduces JANUS, a benchmark designed to evaluate goal-conditioned information distortion in large language models (LLMs) by focusing on the selective treatment of true facts rather than outright falsehoods. The benchmark consists of 160 scenarios across eight domains, comparing neutral and goal-directed conditions to assess how LLMs manipulate factual information to achieve specific objectives. Experiments with 12 LLMs reveal that these models are prone to significant distortions based on framing and incentives, highlighting a critical vulnerability in their communication capabilities.
LLMs exhibit systematic goal-conditioned distortions, revealing a hidden layer of susceptibility to misleading communication that existing benchmarks fail to capture.
LLM deception is often evaluated through direct markers such as fabricated claims, explicit lies, or strategic concealment. However, many real-world misleading communications do not depend on false statements, rather, they arise from selective treatment of true material facts: omitting adverse evidence, softening unfavorable details, emphasizing favorable details, or replacing precise qualifications with vague language. Existing benchmarks largely miss this subtler and arguably more dangerous failure mode. We introduce JANUS, a benchmark for measuring goal-conditioned pragmatic distortion in fact-grounded LLM outputs. Each scenario in our benchmark provides a fixed pool of favorable and adverse facts and compares a neutral condition against a goal-directed condition, such as increasing adoption, enrollment, approval, or support, despite potential harm to directly affected individuals or groups. Because all outputs are constrained to use the same fact pool, JANUS isolates misleading net impressions from hallucination and fabrication. JANUS contains 160 scenarios across 8 domains, with each scenario paired with neutral and goal-conditioned prompts and annotated material facts. Extensive experiments across 12 LLMs reveal consistent goal-conditioned distortions, demonstrating that current models remain sensitive to incentive and framing objectives and lack robust safeguards against selectively misleading communication. We publicly release our corpus and code for future research.
