Search papers, labs, and topics across Lattice.
This paper introduces Buddy, a budget-driven dynamic depth routing framework designed to optimize inference costs in large language models (LLMs) by adaptively selecting Transformer layers based on user-defined compute budgets. By employing a lightweight Decision Module that scores intermediate layers and leveraging a global context source from the first-layer KV cache, Buddy enables real-time adjustments during decoding, improving both efficiency and accuracy. Experimental results demonstrate that Buddy outperforms traditional static pruning methods while maintaining strict budget control and supporting multiple budgets within a single model.
Adaptive layer selection in LLMs can significantly enhance inference efficiency without sacrificing accuracy, achieving better performance than static pruning methods.
Large language models (LLMs) incur high inference cost due to their depth and parameter scale. Depth pruning can reduce latency by skipping redundant Transformer blocks, but existing methods (i) provide limited control under user-specific compute budgets and (ii) typically fix the routing path, failing to adapt as the context grows during decoding. We propose Buddy, a budget-driven dynamic depth routing framework. Buddy uses a lightweight Decision Module to score intermediate layers conditioned on the input and deterministically executes the top-k layers to satisfy a given budget. To support decode-time adaptation, Buddy reuses the first-layer KV cache as a low-overhead global context source and pools it together with the newest token representation before each routing decision. When no explicit budget is provided, an optional Budget Predictor estimates an input-dependent compute level to balance quality and efficiency. Experiments on Llama-family and Qwen models show that Buddy is competitive with strong static pruning baselines and often improves the accuracy-compute trade-off, while uniquely supporting strict budget control, decode-time rerouting, and multiple budgets within a single trained model.
