Search papers, labs, and topics across Lattice.
This paper introduces DepthWeave-KV, a novel token-adaptive cache compression method designed to optimize memory usage during long-context language model inference. By employing cross-layer residual factorization and a token-conditional depth router, the method dynamically allocates reconstruction ranks based on the importance of tokens, significantly enhancing retrieval accuracy while reducing memory bandwidth requirements. The approach achieves an impressive 8.3x reduction in key-value memory usage and maintains high task quality across various benchmarks, demonstrating its effectiveness in real-world applications.
Achieving an 8.3x reduction in memory usage without sacrificing retrieval quality could redefine efficiency standards for long-context language models.
Long-context language model inference is increasingly limited by the memory bandwidth and capacity required to store key-value caches, yet existing compression methods often apply uniform budgets across layers or tokens and degrade retrieval when lexical cues and semantic states require different preservation. We introduce DepthWeave-KV, a token-adaptive cache compression method that factorizes key and value states across neighboring transformer layers using shared low-rank channel bases while retaining lightweight token-specific residuals where attention behavior is sensitive. DepthWeave-KV combines cross-depth residual factorization with a token-conditional depth router that allocates higher reconstruction rank to instruction-bearing and retrieval-critical tokens, and uses calibration-free online error tracking from attention-output probes to adapt compression during generation without retraining the base model. A fused CUDA implementation jointly performs basis lookup, residual dequantization, and attention projection to reduce decode-time memory traffic. Across LongBench, Needle-in-a-Haystack, L-Eval, and long-form QA and summarization benchmarks, DepthWeave-KV achieves near-full-cache task quality with substantially lower memory use, improving average score and retrieval accuracy over prior compressed caches while reaching 8.3x KV memory reduction and 72.8 tokens per second at 64K context.
