Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
We introduce OVAL, an output-aware local page-based retrieval mechanism for long context LLM inference that compactly encodes KV pages by jointly considering key and value structures. Unlike prior methods that focus on attention score or page relevance, OVAL preserves key information crucial for accurate value-weighted attention output without requiring training or additional inference-time statistics. This approach consistently outperforms key-only spectral baselines and achieves competitive performance with state-of-the-art KV cache compression and retrieval methods across various long context tasks, with only modest decoding overhead.
LLM inference cost plummets as a novel output-aware KV cache retrieval method bypasses attention score approximation errors, achieving competitive performance with minimal overhead.
Long context inference with large language models becomes increasingly expensive as attention must operate over an ever growing KV cache. Page sparse attention reduces this cost by representing each KV page compactly and retrieving only a subset for each query. Existing retrieval methods are designed to estimate attention scores or page relevance, but their objectives do not directly account for how approximation errors affect the resulting value weighted attention output. We introduce \method{}, an output aware page encoding derived from the joint structure of keys and values while preserving the key information needed for accurate retrieval. \method{} is training free and requires no additional value dependent statistics at inference time. Once constructed, its stored representation has the same size and decode time scoring cost as a key only spectral representation. Across long reasoning, long context understanding, and long generation benchmarks, \method{} consistently improves over the key only spectral baseline and performs competitively with recent KV cache compression and retrieval methods. On long reasoning benchmarks, it achieves strong avg@\(k\) performance across model benchmark pairs, while matching or surpassing leading baselines on several long context understanding and generation settings with modest decoding overhead. Code is available at \url{https://github.com/Ashkan13776/oval-kv}.