Search papers, labs, and topics across Lattice.
This paper introduces MACRO, a novel framework for dynamic layer routing in Large Language Models (LLMs) that optimizes execution paths without altering model parameters. By modeling layer routing as a context-dependent Markov policy, MACRO effectively supports operations like skipping and repeating layers, leading to significant performance enhancements. The approach yields a 5.0% average accuracy improvement over traditional methods and outperforms the leading dynamic routing technique, Dr. LLM, while drastically reducing route-search time by 9.4 times.
Dynamic layer routing can boost LLM accuracy by 5% without the need for weight updates or expensive search loops.
Standard Large Language Models (LLMs) execute layers sequentially. Dynamic layer routing, i.e. search for a different execution path through layers involving layer repetitions, skips and other moves, can improve performance. Existing routing approaches often require updating model weights, running expensive search loops per test instance, or demand ground-truth labels during inference. In this work, we propose Markov Chain Routing of Transformer Layers (MACRO), a framework that learns task-specific routes over LLM architectures without modifying underlying parameters. MACRO models layer routing as a context-dependent Markov policy conditioned on layer indices, computation budget phases, directional displacements, and operator context, supporting skip, repeat, and residual hidden-state addition operations. The Markov route distribution is updated via feedback on training data and decoded using a top-k Viterbi algorithm to isolate high-probability candidate programs. We evaluate MACRO across diverse reasoning and knowledge benchmarks on multiple open-weight LLMs. MACRO achieves a +5.0% average accuracy improvement over the unrouted baselines, with largest gains on small models. We outperform the best dynamic routing approach Dr. LLM by +7.2%, while reducing route-search time 9.4x (from 14.8 to 1.6 hours). Our code is publicly available at https://github.com/Batorskq/MACRO.
