Search papers, labs, and topics across Lattice.
This paper introduces Active Spatial Guidance (Guidance), a novel training-only objective for Vision Transformers (ViTs) that eliminates the need for injected positional mechanisms by utilizing an auxiliary 2D coordinate-regression loss. By training ViTs without explicit positional injection, the authors demonstrate that spatial organization can be effectively learned from data, leading to significant performance improvements on tasks such as ImageNet-100 classification and ADE20K semantic segmentation. The results indicate that spatial inductive bias can be achieved through training supervision rather than architectural modifications, enhancing robustness and accuracy across various input resolutions.
Spatial inductive bias in Vision Transformers can be learned from data rather than injected, leading to superior performance without architectural constraints.
Vision Transformers (ViTs) commonly rely on injected positional mechanisms to address self-attention's permutation invariance. Motivated by the spatial regularities of natural images, we ask whether spatial organization can be induced from data rather than explicitly injected. Under controlled, matched from-scratch training, we propose Active Spatial Guidance (Guidance), a training-only objective that disables positional injection and applies an auxiliary 2D coordinate-regression loss to the final-layer patch tokens. The guidance head is used only during training and removed for inference; the deployed model consists of a positional-injection-free ViT encoder and the task-specific prediction module. Using DINOv3 ViT backbones, Guidance consistently improves performance on ImageNet-100 classification, ADE20K semantic segmentation, and Hypersim monocular depth estimation, outperforming strong injected baselines such as learned absolute positional embeddings and rotary positional embeddings under identical training protocols. On ImageNet-100, broader comparisons against representative injected positional designs further support Guidance's effectiveness. Guidance also improves robustness under resolution transfer, and multi-resolution training further strengthens accuracy across input sizes. Overall, our results suggest that spatial inductive bias in ViTs need not be architecturally injected, but can be shaped through training-time supervision. The code used for training and evaluation is publicly available in https://github.com/cloudlc/asg.
