Search papers, labs, and topics across Lattice.
The paper introduces SharedLLM, a framework that extends the context window of LLMs by stacking two short-context LLMs—a compressor and a decoder—derived from the same underlying LLM layers via "self-injection." The lower model compresses long inputs into multi-grained representations, which are then selectively passed to the upper model, avoiding full forward passes and cross-attention. Trained on 8K token sequences, SharedLLM generalizes to 128K+ token inputs and achieves comparable or superior performance to baselines on long-context tasks, while also reducing memory footprint and improving inference speed.
Extend your LLM's context window by 16x without retraining from scratch: just "self-inject" a compressed representation from a smaller model into a larger one.
The limited context window of contemporary large language models (LLMs) remains a primary bottleneck for their broader application across diverse domains. Although continual pre-training on long-context data offers a straightforward solution, it incurs prohibitive data acquisition and computational costs. To address this challenge, we propose~\modelname, a novel framework based on multi-grained context compression and query-aware information acquisition. SharedLLM comprises two stacked short-context LLMs: a lower model serving as a compressor and an upper model acting as a decoder. The lower model compresses long inputs into compact, multi-grained representations, which are then forwarded to the upper model for context-aware processing. To maximize efficiency, this information transfer occurs exclusively at the lowest layers, bypassing lengthy forward passes and redundant cross-attention operations. This entire process, wherein the upper and lower models are derived from the same underlying LLM layers, is termed~\textit{self-injection}. To support this architecture, a specialized tree-based data structure enables the efficient encoding and query-aware retrieval of contextual information. Despite being trained on sequences of only 8K tokens, \modelname~effectively generalizes to inputs exceeding 128K tokens. Across a comprehensive suite of long-context modeling and understanding benchmarks, \modelname~achieves performance superior or comparable to strong baselines, striking an optimal balance between efficiency and accuracy. Furthermore, these design choices allow \modelname~to substantially reduce the memory footprint and yield notable inference speedups ($2\times$ over streaming and $3\times$ over encoder-decoder architectures).
