Search papers, labs, and topics across Lattice.
This paper introduces LoFa, a benchmark designed to evaluate the robustness of Large Language Models (LLMs) against logical fallacies through a multi-agent pipeline that pairs factual questions with fallacious arguments. By employing a multi-round debate framework, the study assesses how well LLMs can withstand sustained adversarial persuasion while also proposing a new metric, Logical Fallacy Resistance at k (LFR@k), to quantify this resilience. The findings reveal that LLMs demonstrate varying degrees of vulnerability to different types of fallacies, highlighting the need for targeted improvements in model robustness.
LLMs show significant vulnerability to logical fallacies, with distinct profiles of resilience that could inform future model training strategies.
Large Language Models (LLMs) exhibit strong semantic capabilities, yet their resilience to manipulative linguistic patterns such as logical fallacies remains underexplored. Prior work has primarily examined whether LLMs can identify or classify fallacies, leaving their robustness against fallacious persuasion insufficiently studied. To address this gap, we introduce LoFa (Logical Fallacy), a comprehensive benchmark for evaluating LLM robustness against fallacies. LoFa is constructed through a multi-agent pipeline that pairs factual questions with fallacious arguments, and is accompanied by a multi-round debate framework for assessing model resilience under sustained adversarial persuasion. To disentangle fallacy robustness from a model's inherent knowledge limitations, we further propose Logical Fallacy Resistance at k (LFR@k), a metric that quantifies resistance to fallacious attacks. Experiments show that LLMs exhibit varying levels of robustness across different fallacy types, revealing distinct vulnerability profiles among models.
Tsinghua AI