Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces SALT, a model-agnostic extractive framework that utilizes a salience-aware lexical trie to enhance long-context compression in large language models. By organizing keywords into a trie based on sentence frequency, SALT effectively allocates computational resources across recurring themes, preventing dominant themes from overshadowing less frequent but relevant ones. The approach significantly reduces prefill computation and memory costs while maintaining thematic coverage, thereby improving the efficiency of LLMs in processing longer prompts and multi-turn dialogues.
SALT redefines long-context compression by ensuring thematic diversity is preserved, preventing dominant themes from monopolizing computational resources.
As large language models (LLMs) process increasingly longer prompts, computation and KV-cache memory costs have emerged as major bottlenecks in inference systems. Existing input-level prompt compression methods address this, but rank each sentence by a scalar relevance score, treating the document as an unstructured pool of words and sentences. Under tight budgets, this causes theme collapse, where the dominant theme(s) of a document consumes the budget, discarding less-frequent yet task-relevant themes. Preserving thematic coverage instead requires allocating the budget across recurring themes rather than scoring sentences in isolation. To this end, we propose SALT, a model-agnostic extractive framework that organizes per-sentence keywords into a trie ordered by sentence frequency (SF), a lightweight, reusable proxy for document thematic structure. This trie-based organization smooths memory allocation and prevents dominant themes from monopolizing the budget. Multi-anchor retrieval activates trie nodes labeled by query keywords at any depth, and the trie persists across dialogue turns, supporting multi-turn use without re-encoding the document. By preserving document themes, SALT reduces the prefill computation and memory cost of long-context prompts while remaining composable with KV-cache methods that target decoding-time latency and memory.