Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Soft Mixture-of-Recursions (SoftMoR), a novel approach that enhances Vision Transformers by leveraging recursion to improve representational capacity without a significant increase in parameters. By learning token-wise mixture weights, SoftMoR effectively combines outputs from multiple recursive steps, allowing for better utilization of intermediate representations. The proposed Soft Recursive Vision Transformer (SR-ViT) shows substantial performance gains on vision tasks, achieving a top-1 accuracy of 82.48% on ImageNet-1K with minimal parameter overhead compared to larger models like DeiT-B.
SoftMoR enables Vision Transformers to achieve a 2.65% accuracy boost on ImageNet-1K while using only 27% of the parameters of larger models.
Recent recursive Transformer studies have primarily reused shared parameters across computation steps to construct compact, parameter-efficient models. In this work, we leverage recursion to build effectively deeper Transformers with stronger representational capacity. However, in Vision Transformers, simply increasing recursion depth does not reliably improve performance, as existing recursive approaches do not fully utilize the intermediate representations produced throughout recursive computation. We propose Soft Mixture-of-Recursions (SoftMoR) and its Vision Transformer instantiation, Soft Recursive Vision Transformer (SR-ViT). SoftMoR learns token-wise mixture weights to softly combine outputs from all recursion steps, allowing intermediate representations to be utilized in a learnable and flexible way. Across diverse vision tasks, SR-ViT consistently improves as recursion depth increases with minimal parameter overhead. On ImageNet-1K, increasing recursion depth from 1 to 4 improves SR-ViT-S top-1 accuracy from 79.83% to 82.48% with only 1.7M additional parameters, outperforming the substantially larger DeiT-B while using approximately 27% of its parameters. These results demonstrate that SoftMoR provides a parameter-efficient path to deeper and stronger Vision Transformers through recursion.