Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces ViT-Up, an implicit feature upsampling framework for Vision Transformers that enhances dense prediction tasks by constructing layer-wise queries from intermediate hidden states instead of relying on external image guidance. This approach mitigates issues of feature leakage and blur commonly associated with shallow image encoders, allowing for accurate feature prediction at arbitrary image coordinates. Experimental results show that ViT-Up outperforms existing image-guided upsamplers, achieving significant improvements in metrics such as mIoU and PCK on benchmark datasets like Cityscapes and SPair-71k.
ViT-Up achieves up to 8.09% improvement in dense prediction accuracy by eliminating reliance on external image guidance in Vision Transformers.
Vision Transformers (ViTs) have become a dominant architecture for visual representation learning, providing exceptionally strong and broadly reusable backbone features. However, ViTs are commonly operated on relatively small patch-token grids due to the quadratic cost of global self-attention, which creates a persistent bottleneck for dense prediction tasks such as semantic segmentation and depth estimation. This has motivated the development of task-agnostic feature upsamplers. While recent state-of-the-art methods produce visually sharp dense representations, their reliance on shallow image encoders for guided upsampling can introduce feature leakage, fragmentation, and blur. We introduce ViT-Up, an implicit feature upsampling framework that replaces external image guidance with layer-wise query construction from intermediate ViT hidden states. This enables feature prediction at arbitrary continuous image coordinates while preserving alignment with the backbone feature space. Experiments demonstrate that ViT-Up consistently outperforms state-of-the-art image-guided upsamplers across dense prediction and semantic correspondence. On DINOv3-S+, ViT-Up improves over prior methods by up to +2.07 mIoU on Cityscapes and +4.17 PCK@0.10 on SPair-71k. With the larger DINOv3-B backbone, these gains increase to +3.36 mIoU and +8.09 PCK@0.10, demonstrating that ViT-Up scales favorably with backbone capacity.