Search papers, labs, and topics across Lattice.
This paper introduces RECOM, a contamination-free evaluation dataset designed to assess the validity and discriminative power of automatic metrics in open-ended question answering on Reddit. The authors find that while cosine similarity effectively distinguishes genuine responses from random noise, it fails to rank models accurately, whereas BERTScore shows potential for ranking but suffers from a significant validity gap when controlling for response length. The findings highlight that the validity-discrimination tradeoff is inherent to the metrics themselves, not the models, suggesting a need for dual-axis reporting in metric evaluation.
No automatic metric can effectively balance validity and discriminative power in evaluating LLM-generated responses, revealing a fundamental limitation in current evaluation practices.
Automatic metrics are the default for evaluating LLM-generated text, yet a metric is quietly asked to do two jobs: tell genuine content alignment from surface coincidence (validity), and tell a better system from a worse one (discriminative power). On open-ended, opinion-driven question answering, the two are in tension. We introduce RECOM (Reddit Evaluation for Correspondence of Models), a contamination-free evaluation dataset of 15,000 r/AskReddit questions (September 2025), each paired with its authentic community replies, which postdate every evaluated model's training cutoff. Scoring five open-source LLMs (7--10B) against every reply each metric paired with a random-derangement noise floor we find that no metric does both jobs well. Cosine similarity separates real from random answers (Cohen's $d \approx 2$) but cannot rank the five models ($|d| < 0.1$); BERTScore precision appears to rank the models (raw $|d|$ up to 0.63), but once response length is controlled this collapses to $|d| = 0.09$ and its validity is weak ($d \approx 0.8$, versus cosine's $\approx 2$). Because every metric scores the same outputs, this validity--discrimination tradeoff is a property of the metrics, not the models, and we argue it stems from representation design. Three independent LLM judges reproduce the validity gap and likewise separate the five models only weakly. We recommend reporting metrics on both axes, with an explicit random-baseline floor. RECOM is publicly available at https://anonymous.4open.science/r/recom-D4B0
Stanford HAI