Search papers, labs, and topics across Lattice.
This paper investigates whether high-quality natural language explanations (NLEs) generated by LLMs from XAI outputs actually improve user task performance in time-series energy forecasting. Across five experiments, the authors found that NLEs, while scoring high on quality metrics, do not improve task accuracy and instead inflate user confidence, driven by the mere presence of text. Furthermore, NLEs impair out-of-distribution detection, highlighting a "Quality-Usefulness Gap" and the need to evaluate XAI-to-NLE pipelines based on downstream task performance.
LLM-generated explanations, despite being high-quality, can actually hurt decision-making by increasing confidence without improving accuracy and even masking model failures.
Prior work shows that Large Language Models (LLMs) can transform Explainable AI (XAI) outputs into Natural Language Explanations (NLEs) that score highly on quality metrics such as plausibility, coherence, and comprehensibility. But does explanation quality translate to practical usefulness? We investigate this question in a time-series energy forecasting domain through five controlled experiments (2,730 judgments across 60 test instances), each operationalising a distinct facet of usefulness studied in the XAI literature. Holding NLE quality constant at the high levels established by a prior factorial study, we find that NLEs do not improve task accuracy on any of the five tasks, while inflating self-reported confidence. A placebic control shows that this confidence boost is driven by text presence rather than content. In an out-of-distribution detection task, NLEs reduce the LLM judge's ability to flag unreliable predictions, providing false reassurance that masks model failure. We characterise these findings as the Quality-Usefulness Gap and argue that evaluation of the XAI-to-NLE pipeline must extend beyond text-quality metrics to downstream task performance.
