Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
VFold compresses LLM KV caches by exploiting inter-layer value symmetry, reducing memory footprint without architectural modifications or performance degradation. This symmetry-aware merging strategy achieves significant compression ratios, even when combined with existing techniques like quantization or key pruning. The work identifies substantial underutilized capacity in value caches, offering a practical avenue for extending context windows under memory limitations.
LLM value caches are surprisingly redundant, allowing for significant memory compression through simple symmetry exploitation without hurting inference speed.
While caching key-value (KV) states accelerates Large Language Model (LLM) decoding, this cache can dominate memory usage at long context lengths. One solution is to compress this memory by exploiting inter-layer cache similarities. However, most existing techniques necessitate architectural changes to LLMs and incur substantial overhead. In this work, we propose a symmetry-aware value cache merging strategy that reduces cache memory while avoiding both harmful performance degradation and architectural overhead during decoding. Furthermore, we show that this approach can be exploited alongside existing cache compression techniques, composing with high-ratio quantization or key cache pruning to reach compression ratios that neither method reaches alone, with minimal additional cost. Ultimately, our findings reveal a major source of underutilized capacity in the value cache, offering a simple yet highly effective direction for scaling context windows under memory constraints.