Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces LayerRoute, an adaptive layer skipping mechanism for agentic language models that optimizes compute allocation by selectively skipping transformer blocks based on input type. By integrating lightweight routing and LoRA fine-tuning, LayerRoute enables significant reductions in computational overhead during structured tool calls while maintaining or improving model quality. The approach achieves a 12.91% skip differential, demonstrating that tool calls can skip 15.25% of FLOPs compared to only 2.34% for planning steps, all while using just 1.10M trainable parameters.
LayerRoute reveals that agentic language models can drastically reduce compute costs by selectively skipping layers, enhancing efficiency without sacrificing performance.
Agentic language model systems alternate between two structurally distinct step types: structured tool calls (short, deterministic, low perplexity) and open-ended planning/reasoning steps (long, complex, high perplexity). Despite this heterogeneity, current inference systems apply identical compute to every step. We introduce LayerRoute, a lightweight adapter that learns to selectively skip transformer blocks on a per-input basis. LayerRoute augments each of the 24 transformer blocks in Qwen2.5-0.5B-Instruct with: (1) a per-layer router (~897 parameters, Linear(896,1)) that outputs a hard binary gate via the straight-through estimator, and (2) LoRA adapters (rank 8, ~1.08M parameters) on the Q/K/V/O attention projections. The backbone weights remain frozen. A single end-to-end training pass on agentic data (Hermes, Glaive, GSM8K, Turing) with a gate regularisation term forces the system to discover which blocks are skippable per input type. After 3,000 steps (6.4 minutes on an A100 40GB), LayerRoute achieves a 12.91% skip differential: tool calls skip 15.25% of FLOPs while planning steps skip only 2.34%, using only 1.10M trainable parameters (0.22% of the 494M backbone). Quality improves over the base model due to LoRA adaptation, with perplexity delta of -1.29 on tool calls and -1.30 on planning.