Search papers, labs, and topics across Lattice.
This paper addresses the limitations of traditional evaluation metrics like PR-AUC in the context of semantic caching for LLMs, revealing that high PR-AUC scores do not guarantee effective deployment. The authors introduce two new metrics, Precision-Cache Hit Ratio (P-CHR) AUC and Calibration Retention Rate (CRR), which provide a more accurate assessment of cache performance and the preservation of ranking quality in real-world applications. Their findings indicate that the calibration gap is primarily influenced by the training objective, suggesting that improving model selection for semantic caching requires a focus on calibration rather than just ranking.
High PR-AUC scores can mislead model selection, as they often correlate with poor real-world performance in semantic caching scenarios.
Semantic caching cuts LLM inference costs by serving a cached response to semantically similar queries. Standard practice evaluates these systems using PR-AUC, a metric that only measures how well scores rank and ignores whether they are usable at a fixed threshold. We show this mismatch leads to systematically poor deployment choices, as models with the highest PR-AUC are often the worst in operation. We introduce Precision-Cache Hit Ratio (P-CHR) AUC, a cache-aware metric that measures precision across cache utilization levels, and Calibration Retention Rate (CRR), which captures how much offline ranking quality survives at deployment. We decompose the operational gap between offline and deployed quality into a recoverable calibration component and an irreducible structural component fixed by the dataset's positive rate. Our experiments show that the calibration gap is governed by the training objective rather than data scale, and post-hoc calibration only partially closes it. Ultimately, model selection for semantic caching is a calibration problem, not a ranking one, and measuring it is the first step to closing the gap.
