Search papers, labs, and topics across Lattice.
This paper introduces the Fixed-Point Reasoning Model (FPRM), a Transformer-based architecture designed to enhance compositional reasoning through looped structures. By employing pre-norm layers and residual scaling, FPRM addresses the signal propagation challenges associated with depth in looped architectures, enabling an end-to-end halting mechanism based on fixed-point convergence. The model demonstrates significant effectiveness across various reasoning benchmarks, including Sudoku and Maze tasks, showcasing its ability to adapt computational resources based on task difficulty.
Fixed-point convergence enables adaptive computation in reasoning tasks, allowing models to efficiently tackle complex challenges without unnecessary resource expenditure.
Looped architectures provide an inductive bias toward learning step-by-step procedures for tasks that require compositional reasoning. The number of effective layers reached by looping determines the quality of the solution these models find. Like deep architectures, looped architectures are prone to a signal propagation problem induced by depth as the halting decision is postponed. In this paper, we address this signal propagation issue using pre-norm layers and residual scaling. Building on these architectural modifications, we propose FPRM, a Transformer-based Fixed-Point Reasoning Model that uses fixed-point convergence as an end-to-end halting mechanism in a looped architecture. We show that fixed-point halting allows FPRM to adapt its compute to task difficulty. FPRM is effective on common reasoning benchmarks, namely Sudoku, Maze, state-tracking, and ARC-AGI.
ETH