Search papers, labs, and topics across Lattice.
This paper introduces River-LLM, a training-free framework for token-level early exit in LLMs that addresses the KV cache absence problem. River-LLM uses a KV-Shared Exit River to generate and preserve missing KV cache states, avoiding recomputation or masking. By predicting cumulative KV errors based on state transition similarity, River-LLM achieves 1.71-2.16x speedup on mathematical reasoning and code generation tasks without significant quality loss.
LLMs can achieve up to 2x inference speedup without retraining by intelligently sharing KV cache states during early exit, sidestepping the usual performance bottlenecks.
Large Language Models (LLMs) have demonstrated exceptional performance across diverse domains but are increasingly constrained by high inference latency. Early Exit has emerged as a promising solution to accelerate inference by dynamically bypassing redundant layers. However, in decoder-only architectures, the efficiency of Early Exit is severely bottlenecked by the KV Cache Absence problem, where skipped layers fail to provide the necessary historical states for subsequent tokens. Existing solutions, such as recomputation or masking, either introduce significant latency overhead or incur severe precision loss, failing to bridge the gap between theoretical layer reduction and practical wall-clock speedup. In this paper, we propose River-LLM, a training-free framework that enables seamless token-level Early Exit. River-LLM introduces a lightweight KV-Shared Exit River that allows the backbone's missing KV cache to be naturally generated and preserved during the exit process, eliminating the need for costly recovery operations. Furthermore, we utilize state transition similarity within decoder blocks to predict cumulative KV errors and guide precise exit decisions. Extensive experiments on mathematical reasoning and code generation tasks demonstrate that River-LLM achieves 1.71 to 2.16 times of practical speedup while maintaining high generation quality.
