Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors evaluate learned KV cache compression methods under deployment conditions, revealing that while distilled "Cartridges" excel at document-specific queries, they severely degrade off-context performance through context contamination, factual degradation, and broken instruction-following. To resolve this trade-off, they develop Cartridges++, which preserves general LLM behavior on non-document queries through inference-time cache routing and training-time off-target data mixing. The resulting architectures eliminate off-context derailment with negligible compute overhead while maintaining state-of-the-art compression ratios and on-context document retrieval accuracy.
Distilling long documents into compressed KV caches silently cripples base model capabilities on unrelated prompts, but a lightweight query router or modest off-target data mixing completely eliminates this derailment.
Serving long documents to a Large Language Model (LLM) repeatedly is expensive: computations grow with context length, and the memory footprint of the key-value (KV) cache balloons. Compressed KV (CKV) representations aim to mimic the cache of a document and are typically computed once and for all, ahead of inference time. Methods to obtain CKVs range from drop mechanisms that reduce their number of columns, to learned approaches. Among the latter, Cartridges have emerged as a leading compression method, learning compact KV representations through distillation on relevant Q/A pairs. While existing evaluations focus primarily on whether Cartridges and other CKVs yield approximately similar responses to document-related, on-context queries, we investigate the crucial deployment question of whether they can handle off-context queries, something the native KV representation is particularly good at, thanks to the mechanics of attention. We observe a fundamental trade-off: while Cartridges perform better for on-context queries, heuristic-variants preserve better the original LLM's ability to operate off-context. We measure this through their capability to avoid context contamination in their response, retain general knowledge, and follow instructions. We propose Cartridges++, simple modifications to cartridges that retain off-context abilities at small or negligible cost. The router variant decides at inference time whether the query should use the learned long-context memory, while the data-mixing variant allocates a small fraction of training Q/As to queries outside the reference long document. Our study shows that assessing CKVs on document utility alone can mask substantial degradation in broader model capabilities, yet those issues can be fixed with benign changes to CKV inference or training.