Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces ProgResViT, an input-adaptive Vision Transformer that optimizes inference by progressively processing images at varying resolutions and model widths. By employing a mechanism called Progress-Conditioned Soft Gating (PSG), the model efficiently refines predictions through multiple rounds, starting with low-resolution inputs and only escalating to higher resolutions when necessary. The results demonstrate that ProgResViT achieves superior accuracy-compute trade-offs compared to existing adaptive methods, reaching 84.9% top-1 accuracy with knowledge distillation, thus enhancing the efficiency of image classification tasks.
Achieving 84.9% accuracy with a progressive inference approach, ProgResViT redefines efficiency in Vision Transformers by adapting resolution and width dynamically.
Vision Transformers (ViTs) typically process every image using a fixed input resolution and model width, even though many images can be classified with substantially less computation. We introduce ProgResViT, an input-adaptive ViT that performs inference progressively across multiple rounds. The first round processes a low-resolution image with a narrow subnetwork. Inference terminates when the prediction is sufficiently confident; otherwise, the model reuses the representations produced in the current round and proceeds with a higher-resolution input and a wider subnetwork to refine its prediction. As all rounds share a single backbone, we propose Progress-Conditioned Soft Gating (PSG), which conditions token fusion and layer outputs on the current round, block, and input resolution. On image classification, applying ProgResViT to DeiT yields better accuracy-compute trade-offs than adaptive-width, adaptive-depth, and dynamic-token baselines. With knowledge distillation, a DeiT-based ProgResViT achieves 84.9% top-1 accuracy, slightly exceeding the reported DeiT-III-S accuracy under a comparable evaluation setting. We show that the same design also provides favorable accuracy-compute trade-offs for self-supervised DINO representations and downstream semantic segmentation. Code is available at https://github.com/ds-kiel/ProgResViT.