Search papers, labs, and topics across Lattice.
This study investigates the capacity of transformers to adaptively organize their reasoning processes through a method called latent algorithm routing, which allows them to switch between different solver families based on the underlying data-generating regime. Using the newly introduced ROUTEBENCH diagnostic benchmark, the authors demonstrate that a 306M parameter transformer model can effectively close 80.9% of the oracle-routing gap, achieving a route F1 score of 84.1 across various perturbations and prompts. The findings indicate that while transformers can exhibit route-like behavior, this capability is not universally applicable to all pretrained language models or unrestricted reasoning tasks.
Transformers can achieve 80.9% of the oracle-routing gap through adaptive reasoning, revealing a nuanced capacity for organizing problem-solving strategies based on context.
A central question in the in-context learning literature is whether transformers can organize episode-level adaptation around different inductive-bias families. We study this question in a controlled setting through latent algorithm routing: route-like behavior in which the solver-family preference changes with the latent data-generating regime while prompt form is held fixed, remains stable under nuisance perturbations, and is selectively influenced by targeted activation interventions without large losses in answer quality. We introduce ROUTEBENCH, a diagnostic benchmark whose regimes differentially favor global shrinkage, sparsity, robustness, and locality, operationalized by ridge-like, lasso-like, Huber-like, and kNN-like family representatives. Across dense decoder-only transformers trained from scratch at 44M-612M parameters, a 306M model closes 80.9 percent of the oracle-routing gap and achieves route F1 of 84.1. The effect remains substantial under natural-language renderings, shuffled supports, lexical paraphrases, and a unified four-way routing setting. Stronger adaptive alternatives, including an input-conditioned soft mixture and an unsupervised Gumbel router, narrow the gap but remain below the 306M and 612M models on route F1 and OOD performance. Probe controls and matched activation-patching controls further show that route-relevant internal directions are decodable and functionally involved in solver-family-consistent output behavior. These results provide controlled evidence that dense transformers trained on ROUTEBENCH can develop route-like internal variables, but they do not establish universal routing in pretrained language models or unrestricted natural-language reasoning.
