Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the transferability of token-pruning policies across different tasks in Vision Transformers, specifically focusing on image classification, semantic segmentation, and object detection. By employing controlled probes to analyze pruning criteria without retraining, the authors identify significant differences in how these tasks rank pruning methods and their sensitivity to attention-based pruning. The introduction of Task-Adaptive Pruning (TAP) leverages task-specific registers to optimize token removal, achieving impressive performance improvements on multiple benchmarks while maintaining high throughput.
Task-Adaptive Pruning (TAP) achieves a remarkable 47.0 mIoU on ADE20K and 53.7 box AP on COCO, all while enhancing encoder throughput by over 30%.
Token-pruning policies are usually designed for a single recognition pipeline, but pretrained Vision Transformers are reused across tasks with different spatial demands. We ask which parts of a pruning policy transfer across image classification, semantic segmentation, and object detection. For each pipeline, controlled probes freeze the no-pruning checkpoint and apply a series of parameter-free reduction criteria at one eligible layer at a time without retraining. The probes reveal three differences: segmentation and detection rank the criteria differently, classification is especially sensitive to attention-based pruning in the earliest layers, and the dense tasks prefer opposite recovery endpoints. These findings motivate Task-Adaptive Pruning (TAP). Existing register tokens serve as task-agnostic storage for feature artifacts. TAP instead introduces one task register per task and activates only the current one. Its evolving state ranks tokens, distributes an exact removal budget over depth, and sets the recovery scale for dense features. At a final keep rate of $ρ=0.5$, our jointly adapted model, TAP-J, reaches $47.0$ mIoU at $1.30\times$ encoder throughput on ADE20K and $53.7$ box AP at $1.32\times$ encoder throughput on COCO while remaining competitive on ImageNet-1K.