Search papers, labs, and topics across Lattice.
This paper presents PReM, a novel context-compression framework that optimizes long-context inference by dynamically managing memory through a dedicated layer that learns when to preserve and refresh contextual information. By employing Phase-Separated Refresh Training, PReM aligns memory selection with generation needs, allowing for adaptive context management that enhances reasoning capabilities. Experimental results demonstrate that PReM significantly outperforms traditional compression methods, achieving superior performance under high compression rates while maintaining answer quality and efficiency.
PReM achieves a remarkable balance between context preservation and refresh, outperforming existing methods even at 32x compression rates.
Efficient long-context inference is not only about reducing memory cost, but also about keeping useful contextual evidence accessible as generation proceeds. However, existing compression-oriented approaches, such as key-value (KV) cache compression and context compression, often either make an early decision about which contextual information to keep or rely on an external compressor. Such designs make it difficult to adapt the compressed context to the evidence needed by later reasoning steps. This paper introduces PReM (Preserve and Refresh Memory), a context-compression framework that maintains the long context as the model's internal layer-wise KV memory and learns what to preserve and when to refresh it. Specifically, PReM uses a dedicated memory layer to make memory-selection decisions, and a special memory tokento trigger refreshes during generation. To train this behavior, PReM introduces Phase-Separated Refresh Training, aligning memory selection with memory-conditioned generation while preserving continuity across refreshes. Experiments with 32K-token contexts show that PReM outperforms strong baselines under both 16x and 32x compression, while maintaining a favorable balance between answer quality and inference efficiency.
