Search papers, labs, and topics across Lattice.
The paper introduces DarkPatterns-LLM, a novel benchmark dataset and diagnostic framework for evaluating manipulative content in LLM outputs across seven harm categories, addressing the limitations of existing binary-labeled safety benchmarks. The framework employs a four-layer analytical pipeline (MGD, MSIAN, THP, DCRA) for fine-grained assessment. Evaluation of state-of-the-art models reveals significant performance disparities (65.2\%--89.7\%) and consistent weaknesses in detecting autonomy-undermining patterns, highlighting the need for improved manipulation detection in LLMs.
LLMs struggle to detect subtle autonomy-undermining manipulation tactics, with performance varying by up to 25% across harm categories, according to a new benchmark.
The proliferation of Large Language Models (LLMs) has intensified concerns about manipulative or deceptive behaviors that can undermine user autonomy, trust, and well-being. Existing safety benchmarks predominantly rely on coarse binary labels and fail to capture the nuanced psychological and social mechanisms constituting manipulation. We introduce \textbf{DarkPatterns-LLM}, a comprehensive benchmark dataset and diagnostic framework for fine-grained assessment of manipulative content in LLM outputs across seven harm categories: Legal/Power, Psychological, Emotional, Physical, Autonomy, Economic, and Societal Harm. Our framework implements a four-layer analytical pipeline comprising Multi-Granular Detection (MGD), Multi-Scale Intent Analysis (MSIAN), Threat Harmonization Protocol (THP), and Deep Contextual Risk Alignment (DCRA). The dataset contains 401 meticulously curated examples with instruction-response pairs and expert annotations. Through evaluation of state-of-the-art models including GPT-4, Claude 3.5, and LLaMA-3-70B, we observe significant performance disparities (65.2\%--89.7\%) and consistent weaknesses in detecting autonomy-undermining patterns. DarkPatterns-LLM establishes the first standardized, multi-dimensional benchmark for manipulation detection in LLMs, offering actionable diagnostics toward more trustworthy AI systems.
