Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces RPCBench, a benchmark designed to evaluate the ability of large language models (LLMs) to critique flawed premises in recommendation requests, addressing a significant gap in existing recommender and error-detection benchmarks. By providing evidence-grounded test instances across five domains and ten types of premise failures, the authors systematically assess LLMs on proactive detection, error localization, and handling strategies. The findings reveal that proactive detection is a critical challenge, particularly for underspecified premises, and highlight the importance of information density and reasoning length in critique quality.
Proactive detection of flawed premises in recommendation requests is a major bottleneck for LLMs, with performance peaking at intermediate reasoning lengths.
Large language models are increasingly used as interactive recommender assistants. Their evaluation should therefore go beyond plausible item recommendation and test whether they can recognize flawed recommendation requests. Existing recommender benchmarks mainly assess ranking, generation, or preference satisfaction, while existing error-detection benchmarks are usually not grounded in recommendation-specific user and candidate evidence. To address this gap, we introduce RPCBench, a benchmark for evaluating Recommender-Premise Critique: the ability to detect, diagnose, and properly handle faulty premises in natural-language recommendation requests. RPCBench contains evidence-grounded test instances from five recommendation domains and covers ten types of premise failures. Each instance provides a visible recommendation context and a corrupted user query. We further design a fine-grained evaluation framework that measures proactive detection, error localization, post-detection handling strategy, and evidence faithfulness. Through a systematic evaluation of 11 LLMs, we find that proactive detection is the main bottleneck in Recommender-Premise Critique, and models perform worst on underspecified-premise errors. We also observe that target-critical information density matters more than redundant evidence, and that longer reasoning does not monotonically improve critique quality: performance peaks at intermediate reasoning length, while overly long reasoning is accompanied by an overthinking penalty. The code is available at https://github.com/ZhongruChen/RPCBench.