Search papers, labs, and topics across Lattice.
This study systematically evaluates various eviction policies for semantic caches in large language models (LLMs), including FIFO, LRU, LFU, and others, across different workloads, cache capacities, and encoders using the CLEVER framework. The findings reveal that no policy significantly outperforms LFU, with alternatives like FIFO and streaming SISO lagging by as much as 8.67 percentage points under tight capacity conditions. Additionally, the research highlights a critical issue with the evaluated operating point, where only a small percentage of hits are deemed answer-substitutable, suggesting that establishing answer validity is crucial before optimizing eviction policies.
LFU emerges as the clear champion among eviction policies, with alternatives failing to deliver meaningful improvements in cache performance.
Semantic caches reuse an LLM response when the incoming query embedding lies near a cached query, but proposed eviction policies have rarely been compared under one protocol. Using CLEVER, we evaluate FIFO, LRU, LFU, ARC, GDSF, a single-pass streaming adaptation of SISO, and a semantic-redundancy policy across three ordered, deduplicated query corpora, three cache capacities, and two encoders. No evaluated policy improves on LFU by more than 0.041 percentage points in any of the eighteen settings. Replacement is not irrelevant: FIFO and streaming SISO trail LFU by as much as 8.67 and 8.55 points, respectively, at tight capacity. We explain the missing upside with a conditional packing result. Under exact lookup and insert-on-miss, a newly inserted entry cannot have a resident neighbor within the hit radius, so a geometry-aware eviction rule receives little new redundancy signal. A separate audit exposes a larger problem with the evaluated operating point. At MiniLM's median nearest-neighbor threshold, only 2.1-3.9% of sampled LMSYS and QQP hits are judged answer-substitutable, reducing raw hit rates of 51-60% to quality-adjusted rates of 1.1-2.2%. The cross-encoder study further shows that thresholds do not transfer between embedding models. LFU is the strongest simple default in this protocol; deployment decisions should first establish answer validity and then test sub-point policy differences with exact search.
