Search papers, labs, and topics across Lattice.
This paper introduces PREDACTBENCH, a benchmark designed to evaluate dialogue agents that operate with noisy tool outputs in high-stakes decision-making contexts, specifically focusing on education. The authors develop novel metrics, Relative AI-Reliance (RAIR) and Relative Self-Reliance (RSR), to assess how well these models support human decision-makers without fostering over-reliance on inaccurate suggestions. Evaluations across 13 state-of-the-art LLMs reveal that current models inadequately provide the necessary transparency to educators, highlighting a critical gap in AI-assisted decision support systems.
When faced with noisy tools, state-of-the-art LLMs fail to prevent educators from over-relying on incorrect suggestions, exposing a significant flaw in AI decision support.
Large Language Models (LLMs) are increasingly deployed in task-oriented dialogue systems that support multi-step decision-making in high-stakes domains such as education, healthcare, and finance. However, existing benchmarks typically assume perfectly accurate tool outputs, overlooking the reality that deployed systems must operate with noisy tools and human decision-makers whose trust in the agent is itself uncertain. Such conditions are common in practice, for example, a clinician using a diagnostic prediction tool or an advisor relying on a model that forecasts student outcomes from historical records. We introduce PREDACTBENCH, a benchmark for evaluating dialogue agents paired with statistically imperfect tools, using education as a measurable testbed where ground truth outcomes and clear intervention decisions are available. First, we build a benchmark for AI-assisted human decision-making, where the AI uses noisy predictors to help guide a user. Second, we introduce episode-level Relative AI-Reliance (RAIR) and Relative self-reliance (RSR) metrics, extending prior trust calibration framework to multi-turn dialogue. Third, we evaluate 13 state-of-the-art closed and open source LLMs on two educational datasets, OULAD (real assessment trajectories from the UK Open University) and PREDACT-CS (60 courses with real final grade outcomes and synthetically generated weekly score trajectories), alongside a human study with instructors and teaching assistants. We find that when tools are noisy, SOTA models are supposed to provide visibility to teachers so that they do not over-rely on wrong suggestions or hallucinations, but current models fail to do that. We offer PREDACTBENCH to help build better LLMs as AI decision support systems to help teachers.
