Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces Mutual Pair Merging (MPM), a training-free token aggregation module for vision transformers that merges mutual nearest-neighbor pairs in cosine space. This approach generates a merge map to facilitate efficient, gather-based reconstruction of dense features before the decoder, enabling compatibility with existing segmentation heads. Benchmarking on NVIDIA H100 and Raspberry Pi 5 demonstrates that MPM achieves significant latency reductions (up to 60% on Raspberry Pi 5) and throughput increases (up to 20% on H100 with FlashAttention-2) with minimal mIoU drop for semantic segmentation.
Achieve up to 20% higher throughput on H100 GPUs for semantic segmentation by simply merging mutually similar tokens in ViTs *without* retraining.
Decreasing sequence length is a common way to accelerate transformers, but prior token reduction work often targets classification and reports proxy metrics rather than end-to-end latency. For semantic segmentation, token reduction is further constrained by the need to reconstruct dense, pixel-aligned features, and on modern accelerators the overhead of computing merge maps can erase expected gains. We propose Mutual Pair Merging (MPM), a training-free token aggregation module that forms mutual nearest-neighbor pairs in cosine space, averages each pair, and records a merge map enabling a gather-based reconstruction before the decoder so that existing segmentation heads can be used unchanged. MPM introduces no learned parameters and no continuous compression knob (no keep-rate or threshold). The speed-accuracy trade-off is set by a discrete insertion schedule. We benchmark end-to-end latency on an NVIDIA H100 GPU (with and without FlashAttention-2) and a Raspberry Pi 5 across standard segmentation datasets. On ADE20K, MPM reduces per-image latency by up to 60% for ViT-Tiny on Raspberry Pi 5, and increases throughput by up to 20% on H100 with FlashAttention-2 while keeping the mIoU drop below 3%. These results suggest that simple, reconstruction-aware, training-free token merging can translate into practical wall-clock gains for segmentation when overhead is explicitly accounted for.