Search papers, labs, and topics across Lattice.
This paper introduces MergeOver, a novel post-training method that integrates Token Merging into the Sliced Recursive Transformer framework to address the challenges of high parameter counts and computational complexity in Vision Transformers. By employing an Unmerge tracking stack and a stage-wise single-shot schedule, MergeOver achieves significant reductions in peak activation memory and latency on resource-constrained devices like the Raspberry Pi 5, while maintaining competitive performance on ImageNet-1K. The approach demonstrates that it is possible to effectively combine token merging with recursive weight-sharing without the need for costly retraining, yielding a practical solution for deploying ViTs in edge environments.
MergeOver reduces peak activation memory by over 37% while maintaining competitive accuracy, making it a game-changer for deploying Vision Transformers on edge devices.
Vision Transformers (ViTs) demonstrate exceptional performance in computer vision but suffer from large parameter counts and quadratic computational complexity, severely limiting their deployment on resource-constrained edge hardware. While recursive weight-sharing reduces parameter counts and token merging mitigates computational and memory bottlenecks, integrating these two paradigms without costly retraining is non-trivial, leaving this intersection largely unexplored. We propose MergeOver, a post-training approach that integrates Token Merging (ToMe) into the recursively weight-shared Sliced Recursive Transformer (SReT). Through an Unmerge tracking stack, constraint-safe merge-rate adjustment, and synchronised token-mass tracking across spatial permutations, MergeOver resolves the spatial and merging constraints of this integration. We further employ a stage-wise single-shot schedule that performs token reduction at the first block of each stage and maintains a fixed sequence length throughout its subsequent recursive iterations. Benchmarked on ImageNet-1K, our selected configuration reduces top-1 accuracy by 1.47 percentage points. On the GPU, it reduces peak activation memory by 37.3% and 38.4% at batch sizes 1 and 16, while throughput decreases by 21.7% at batch size 1 but increases by 21.7% at batch size 16. On a Raspberry Pi 5 (ARM CPU), it reduces latency by 2.4% and 17.6% at batch sizes 1 and 16. These results show that MergeOver can recover a meaningful part of the throughput and memory cost that recursive weight-sharing introduces, without retraining, and provides a baseline for combining token merging with hierarchical recursive transformers.
