Search papers, labs, and topics across Lattice.
This paper introduces a behavior-grounded LLM judge that enhances the evaluation of search engine result pages (SERPs) by incorporating historical user interaction data through Query-Relevance-Impressions (QRI) cards. By grounding relevance judgments in empirical user behavior, the approach significantly improves alignment with actual user preferences, achieving a 5% increase in Spearman rank correlation and a 15% boost on a multilingual dataset. The method demonstrates practical effectiveness in real-world applications, as evidenced by its superior performance in A/B testing compared to traditional LLM evaluation methods.
Grounding LLM evaluations in historical user behavior can boost relevance judgment accuracy by over 15%, making them more aligned with actual user preferences.
Large-scale search systems evolve faster than human quality assurance can scale, especially for long-tail intents and multilingual queries. LLM-as-a-judge approaches provide a scalable alternative for evaluating the relevance of search engine result pages (SERPs), but judgments based solely on semantic similarity or world knowledge can drift from actual user preferences, particularly for ambiguous queries. We introduce a behavior-grounded LLM judge that augments each SERP item with a lightweight and auditable behavioral prior in the form of a Query-Relevance-Impressions (QRI) card. Each card summarizes how users have historically interacted with similar queries and results, providing compact empirical evidence that the judge can cite to resolve ambiguity and make more consistent relevance judgments while still relying on semantic reasoning. In a large-scale music search evaluation at Spotify, using relevance estimates derived from historical user interactions across 6,000 recomposed SERPs, the behavior-grounded judge achieves stronger alignment with user preferences, improving Spearman rank correlation by approximately 5% overall and yielding a 91% relative improvement on disagreement cases. On a multilingual human-judged dataset spanning five languages, grounding further increases correlation with human relevance judgments by 15%. Importantly, when evaluated against outcomes from a live A/B test, the grounded judge shows consistently higher alignment with the observed winning model. While absolute alignment remains moderate, these findings demonstrate that lightweight behavioral grounding can improve the reliability and practical usefulness of LLM-based evaluation in real-world search systems.
