Search papers, labs, and topics across Lattice.
To address the myopic nature of standard dynamic layer-skipping鈥攚hich conditions decisions solely on local hidden states despite routing being inherently path-dependent鈥攖he authors introduce HeRo, a framework that maintains an explicit routing memory across model depth. Built with a depth-wise linear-attention mechanism that accumulates preceding routing decisions and residual updates, HeRo trains only lightweight routers and adapters while keeping pretrained LLM weights frozen. Across Llama 2 and 3.1 backbones, the approach outperforms ten dynamic routing baselines, notably allowing Llama 3.1-8B to bypass 26.87% of its parameters while achieving 100.24% of dense performance across seven benchmarks.
Dynamic layer-skipping fails on complex reasoning tasks because standard routers suffer from depth amnesia, but tracking execution history across layers lets Llama 3.1-8B bypass over a quarter of its parameters with zero performance loss.
Dynamic layer routing reduces the inference cost of Large Language Models (LLMs) by learning to skip layers for individual tokens. Existing methods, however, treat each routing decision as a local operation conditioned solely on the current hidden state which is a formulation that overlooks the sequential, path-dependent nature of routing across depth: earlier decisions shape the representations seen by downstream routers, and the layer-usage objective couples all decisions jointly. We propose History-Aware Routing (HeRo), a dynamic routing framework that resolves this mismatch by introducing a router memory mechanism to maintain an explicit routing state across model depth. The memory is constructed via linear attention, incrementally aggregating preceding routing scores and their induced residual updates into a compact history representation. At each routed layer, the router conditions jointly on this accumulated state and the current hidden representation to select the executed branch. Instantiated for token-wise FFN routing, HeRo trains only lightweight routers and adapters on a frozen backbone, requiring no modification to pretrained parameters. Across Llama 3.1-8B, Llama 2-7B, and Llama 2-13B, HeRo consistently achieves the highest aggregate performance retention among ten baselines. On Llama 3.1-8B, it bypasses 26.87% of model parameters while achieving 100.24% of dense model performance across seven benchmarks, and retains 97.01% while bypassing 38.82% of model parameters under a tighter computation budget. Ablation studies confirm that removing routing history consistently degrades performance, most notably on multistep reasoning and code generation, validating that explicit routing memory enables more accurate and adaptive dynamic routing than solely conditioning on hidden state.
