Search papers, labs, and topics across Lattice.
This paper introduces an architectural enhancement called recirculation for foundation models, which significantly reduces perplexity and improves accuracy in generation and reasoning tasks without adding latency during generation. By implementing a specific form of recurrence, recirculation allows models to function as dynamical systems that effectively track belief states, distinguishing it from existing methods like chain-of-thought and depth-recurrence techniques. The adaptive variant of recirculation achieves a 23% reduction in perplexity and a 21% increase in accuracy on the GSM8k dataset, demonstrating the potential for training-free architectural evolution based on model properties.
Recirculation enables foundation models to achieve a 23% reduction in perplexity and a 21% increase in accuracy without any added generation latency.
We describe an inference-time architectural enhancement for off-the-shelf foundation models that markedly reduces perplexity and boosts accuracy across generation and reasoning tasks. Our approach incurs essentially no additional latency during generation, though it requires serial processing in the prefill phase. Motivated by the fundamental limitation that state updates in feedforward transformers are bounded by model depth, our technique, recirculation, introduces a specific form of recurrence that allows the model to act as a dynamical system and track belief states. We distinguish this technique from chain-of-thought computation---which is better reserved for complex inferences rather than basic state tracking---as well as from popular depth-recurrence techniques (looping) and the costly training of recurrent transformers. We also propose and evaluate an adaptive variant of recirculation which requires only light tuning of hyperparameters while freezing the original model weights. Relative to the off-the-shelf baseline, adaptive recirculation achieves remarkable gains on the Gemma3 family, including a 23% reduction in perplexity on a suite of datasets, a 21% increase in accuracy on GSM8k, and reliable improvements in accuracy on other downstream tasks. Our training-free approach succeeds by leveraging the model itself to inform architectural modifications, suggesting a route to architectural evolution guided by a trained network's properties rather than forced, arbitrary design choices.
