Search papers, labs, and topics across Lattice.
The paper investigates whether the Tiny Recursive Model (TRM) refinement mechanism, previously shown to be effective in ARC-AGI, can be successfully integrated into autoregressive models. They propose the Autoregressive TRM and compare it against a suite of models that gradually transform a standard Transformer to an Autoregressive TRM, controlling for block design, token stream, and next-token prediction objective. Surprisingly, they find that while some two-level refinement baselines perform well, the full Autoregressive TRM architecture does not provide reliable performance gains in compute-matched character-level algorithmic tasks.
Despite the buzz around Tiny Recursive Models, directly adapting their refinement mechanism into autoregressive architectures yields no reliable performance boost, suggesting the original TRM's success may stem from other factors.
Tiny Recursive Models (TRMs) have recently demonstrated remarkable performance on ARC-AGI, showing that very small models can compete against large foundation models through a two-step refinement mechanism that updates an internal reasoning state $z$ and the predicted output $y$. Naturally, such refinement is of interest for any predictor; it is therefore natural to wonder whether the TRM mechanism could be effectively re-adopted in autoregressive models. However, TRMs cannot be simply compared to standard models because they lack causal predictive structures and contain persistent latent states that make it difficult to isolate specific performance gains. In this paper, we propose the Autoregressive TRM and evaluate it on small autoregressive tasks. To understand its efficacy, we propose a suite of models that gradually transform a standard Transformer to a Tiny Autoregressive Recursive Model in a controlled setting that fixes the block design, token stream, and next-token objective. Across compute-matched experiments on character-level algorithmic tasks, we surprisingly find that there are some two-level refinement baselines that show strong performance. Contrary to expectations, we find no reliable performance gains from the full Autoregressive TRM architecture. These results offer potential promise for two-step refinement mechanisms more broadly but caution against investing in the autoregressive TRM-specific model as a fruitful research direction.
