Search papers, labs, and topics across Lattice.
This paper addresses the Knowledge Selection-Runtime Consumption (KSRC) gap in large language models (LLMs), where the rich structured knowledge used for reasoning is lost in the serialization process, leading to inefficient memory usage and increased latency. The authors introduce Knowledge Access Planning (KAP), which transforms structured knowledge signals into a universal intermediate representation that optimizes key-value access during LLM serving without altering the underlying model. Their implementation, GraphSpec, demonstrates that KAP can significantly reduce memory consumption while maintaining answer quality across long-context question-answering tasks, achieving a reduction in KV access to just 5.5% of the source state at 128K tokens.
KAP redefines LLM serving by turning knowledge selection into an efficient execution strategy, slashing memory access needs while preserving output quality.
Modern LLM systems increasingly rely on knowledge-selection processes that produce high-value structured priors, such as ranked evidence, graph topology, multimodal alignment, and confidence signals. Yet LLM serving remains fundamentally oblivious to this rich structure: once such signals are serialized into a prompt, the backend observes only a flat token sequence, forcing dense and uniform consumption of the full key-value (KV) state during decoding. We term this architectural mismatch the Knowledge Selection-Runtime Consumption (KSRC) gap: richer contexts enlarge the full-prompt KV footprint and decode-time memory traffic, increasing latency and degrading throughput even when reasoning depends on only a small fraction of the context. To bridge the gap, we propose Knowledge Access Planning (KAP), a paradigm-shifting execution abstraction that elevates structured knowledge priors from passive prompt-construction hints into first-class physical execution artifacts. KAP establishes a universal intermediate representation (IR)-the runtime access plan-which compiles structured knowledge signals to govern physical KV access without altering logical prompt semantics, model weights, or training procedures. Through this IR, KAP shifts LLM serving from token-aware context consumption to plan-driven, knowledge-aware runtime consumption. We instantiate KAP with GraphSpec, a compiler-executor realization connecting structured knowledge selection to an LLM serving backend. We derive a phase-boundary model for the positive-speedup regime of plan-guided execution. Across 4K-128K long-context QA workloads, GraphSpec maintains answer quality comparable to full-context decoding while decoupling physical KV consumption from prompt length, reducing proposal-time KV access to 5.5% of source KV state at 128K, and fundamentally shifting the scaling trajectory of long-context generation.
