Search papers, labs, and topics across Lattice.
The paper introduces STAR, a novel autoregressive pretraining method for Vision Mamba that uses separators to demarcate images in a sequence, enabling the model to process significantly longer input sequences. By inserting identical separators before each image, the method effectively quadruples the input sequence length without altering image dimensions. Experiments show that a STAR-B model pretrained with this technique achieves 83.5% accuracy on ImageNet-1k, demonstrating the benefits of long-sequence pretraining for Vision Mamba.
Vision Mamba's ImageNet accuracy jumps to 83.5% thanks to a simple trick: adding separator tokens to enable pretraining on 4x longer sequences.
The state space model Mamba has recently emerged as a promising paradigm in computer vision, attracting significant attention due to its efficient processing of long sequence tasks. Mamba's inherent causal mechanism renders it particularly suitable for autoregressive pretraining. However, current autoregressive pretraining methods are constrained to short sequence tasks, failing to fully exploit Mamba's prowess in handling extended sequences. To address this limitation, we introduce an innovative autoregressive pretraining method for Vision Mamba that substantially extends the input sequence length. We introduce new \textbf{S}epara\textbf{T}ors for \textbf{A}uto\textbf{R}egressive pretraining to demarcate and differentiate between different images, known as \textbf{STAR}. Specifically, we insert identical separators before each image to demarcate its inception. This strategy enables us to quadruple the input sequence length of Vision Mamba while preserving the original dimensions of the dataset images. Employing this long sequence pretraining technique, our STAR-B model achieved an impressive accuracy of 83.5\% on ImageNet-1k, which is highly competitive in Vision Mamba. These results underscore the potential of our method in enhancing the performance of vision models through improved leveraging of long-range dependencies.
