Search papers, labs, and topics across Lattice.
This paper introduces SemPIC, a novel approach that enhances long-context retrieval by utilizing position-independent caching to improve the efficiency of key-value (KV) storage in language models. By employing a LoRA-enabled Writer for behavioral distillation, SemPIC adapts document representations while maintaining the pretrained decoder as a Reader, effectively addressing attention deviations and residuals in KV states. The method achieves a mean micro-F1 score of 0.60 across multiple tasks, demonstrating significant improvements over existing KV caching techniques and nearing the performance of full recompute methods.
SemPIC boosts long-context retrieval performance by 13% over traditional KV caching, nearing the efficiency of full recompute without sacrificing model integrity.
Long-context retrieval and agentic workloads repeatedly reuse the same documents under changing instructions, histories, and document orders. Prefix caching cannot exploit this reuse, while position-independent caching (PIC) remains unreliable because independently compiled KV states lack the future context in which they will be consumed. Our diagnostics show that a learned boundary-conditioned baseline sharply reduces attention deviation near reusable-block boundaries but leaves interior and task-level residuals, motivating adaptation of the document representation itself. We present \emph{SemPIC}, which trains a LoRA-enabled Writer to compile native per-layer document KVs through behavioral distillation while retaining the pretrained decoder as an unchanged Reader. Adaptation is confined to offline cache construction, preserving the standard KV interface and cache-hit decoding path. We further introduce KV Gradient Checkpointing, which reduces peak training memory without severing gradients through cached KVs. Across three models and four tasks, SemPIC raises mean micro-F1 over KV Packet from 0.53 to 0.60, approaching Full Recompute at 0.62.
