Search papers, labs, and topics across Lattice.
This study benchmarks token compression and structural pruning methods for Vision Transformer (ViT) segmentation under high compression and corrupted input scenarios. The findings reveal that while token compression is effective at low levels, it suffers significant performance degradation under severe compression, whereas structural pruning maintains stability and robustness. A novel prune-then-merge strategy combining moderate token compression with structural pruning is proposed, yielding superior accuracy and robustness at high compression levels, thus enhancing deployment feasibility.
Token compression falters under severe constraints, while structural pruning offers a more stable solution for robust ViT segmentation.
Vision Transformers (ViTs) are strong backbones for semantic segmentation, but their computational cost limits deployment. Recent token compression methods for efficient transformer-based segmentation reduce this cost by decreasing the number of tokens. However, existing evaluations primarily focus on low-to-moderate compression, leaving their behavior under aggressive compression and corrupted inputs unclear. Meanwhile, structural pruning provides an orthogonal route to efficiency by removing redundant components in the ViT architecture, but is rarely compared to token compression under a unified protocol. To bridge this gap, we benchmark representative token compression and structural pruning methods for ViT-based semantic segmentation under matched FLOPs on ADE20K and Cityscapes, together with their common-corruption variants ADE20K-C and Cityscapes-C. Our results reveal a consistent trend on both clean and corrupted inputs: token compression is highly effective at mild reductions but degrades sharply when compression becomes severe, consistent with substantial information loss from overly aggressive token reduction. In contrast, structural pruning exhibits a smoother degradation curve and is more stable at high compression. Motivated by these findings, we study a prune-then-merge pipeline that applies moderate token compression on top of a moderately pruned backbone. At comparable FLOPs, this combined strategy consistently achieves a better accuracy-robustness trade-off at high compression, offering a practical recipe for deployment-oriented ViT segmentation. Code is available at https://github.com/phatnguyencs/vit-seg-compression.
