Search papers, labs, and topics across Lattice.
This paper introduces the LLM-as-a-Judge framework to address the limitations of traditional Top-K metrics in offline recommendation evaluation, which often rely on biased user feedback and lack interpretability. By employing a semantic proxy that captures true user preferences from textual behaviors, the framework allows for more flexible and accurate evaluations. The results show that the LLM Judge not only enhances reliability but also provides explicit rationales for its judgments, significantly improving the explainability of recommendation assessments.
Traditional Top-K metrics fail to capture true user preferences, but the LLM-as-a-Judge framework offers a semantic approach that enhances both reliability and explainability in recommendation evaluations.
Recommendation evaluation plays a crucial role in guiding the refinement and deployment of recommender systems. Most existing trials rely on offline evaluation using Top-K metrics computed over holdout user behaviors. However, we identify two fundamental limitations that undermine their ability to deliver reliable and explainable evaluations. Regarding reliability, offline evaluation treats observed user feedback as a proxy of true preferences and enforces rigid ID matching between the proxy and recommendation. In practice, feedback collections are inherently shaped by incomplete and biased item exposure, leading to distorted and unreliable assessments. Regarding explainability, Top-K metrics only establish numerical scores without offering meaningful insights to support them, thereby reinforcing the black-box nature of offline evaluation. In this paper, we propose a reliable and explainable LLM-as-a-Judge framework for offline recommendation evaluation. To enhance reliability, we introduce a semantic proxy from user textual behaviors to represent their true preferences. This proxy allows for more flexible matching between preferences and recommendations in the semantic space, rather than depending on the holdout feedback. To ensure explainability, the LLM Judge adopts a reasoning-then-scoring process to generate relevance judgments along with explicit rationale. Finally, we aggregate the individual scores into global Top-K metrics to quantify overall recommendation quality, and provide justification for each preference hit or miss. Extensive experiments demonstrate that the LLM Judge achieves solid reliability, explainability, and robustness in evaluation.
