Search papers, labs, and topics across Lattice.
This paper empirically characterizes the performance of single-block recurrent Vision Transformers (bViT) against standard Vision Transformers (ViTs) under various training and inference regimes. The authors investigate the conditions under which recurrence outperforms independently parameterized depth, the role of solver order in training residual recurrent blocks via ODEs, and the trade-offs between robustness and nominal accuracy. Their findings indicate that while standard ViTs excel under FLOP constraints, bViTs provide a superior accuracy-to-parameter ratio when memory is limited, highlighting the nuanced performance dynamics of these architectures.
Recurrent Vision Transformers can outperform standard models in accuracy-to-parameter trade-offs when memory constraints are prioritized, challenging conventional wisdom about architectural efficiency.
Vision Transformers (ViTs) achieve strong image-recognition performance, but their parameter count grows linearly with depth when each block is independently parameterized. Single-block recurrent ViTs (bViT) remove this growth by repeatedly applying one shared block. Rather than proposing a new architecture, we fix a bViT and provide a controlled empirical characterization of three training and inference regimes under a common CIFAR-100 protocol, asking: (i)~when does recurrence beat independently parameterized depth---at matched FLOPs or at matched parameter memory? (ii)~when a residual recurrent block is trained through an ODE solver, does solver order act as numerical refinement or as an architectural bias? and (iii)~what does robustness beyond the training horizon cost in nominal accuracy? We find that standard ViTs remain preferable when FLOPs are the primary constraint, whereas recurrent ViTs offer a better accuracy--parameter trade-off under memory constraints. Consistent with the standard view of residual networks as Euler discretizations of ODEs, the continuous-time analogue of a residual recurrent block is the state-subtracted vector field $\dot{z}=F_胃(z)-z$; although known in principle, this distinction is easy to violate when the block is wrapped as a black-box vector field, and we qualify the cost at few accuracy points. Because the vector field is learned jointly with the solver, higher-order solvers act as a solver-induced architectural bias rather than a numerical-accuracy improvement, and their gains are not uniform. Finally, stage-wise deep supervision traces an accuracy--robustness frontier: it does not improve nominal accuracy, but degrades gracefully far beyond the training horizon, where naive recurrence collapses to near-random performance.
