Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the instability in Vision Transformers (ViTs) caused by fixed patch grids, particularly how varying patch partitions can affect token evidence near pixel boundaries. The authors introduce Phase Marginalization, a training-free method that evaluates and aligns structured patch-grid phases to enhance dense outputs in the original image coordinate system. Their key finding reveals that Uniform Phase Marginalization with K = 4 outperforms the traditional K = 1 baseline across various tasks, providing a significant compute-matched advantage in segmentation and depth estimation tasks.
Phase Marginalization reveals that optimizing patch-grid phase can lead to substantial improvements in dense prediction tasks without the need for additional training.
Vision Transformers operate on fixed patch grids, which can introduce phase-dependent instability for dense prediction: changing the patch partition can change the token evidence available to a pixel, especially near boundaries. We formalize patch-grid phase as a nuisance variable and propose Phase Marginalization, a post-hoc marginalization method that evaluates structured patch-grid phases, inverse-aligns dense outputs, and aggregates them in the original image coordinate system. The central variant, Uniform Phase Marginalization with K = 4, is training-free and improves over the canonical K = 1 baseline across measured segmentation, depth, and local matching settings. In a controlled Cityscapes experiment, Uniform Phase Marginalization provides a modest compute-matched advantage over generic shift-based four-forward test-time augmentation (TTA) (+0.31 mean Intersection-over-Union over the strongest tested generic row). A scaling study further shows that K = 4 is a practical cost-accuracy trade-off: K = 8 is essentially unchanged and K = 16 adds little accuracy at much higher latency. These results position patch-grid phase as a measurable nuisance variable and Phase Marginalization as a simple diagnostic and post-hoc marginalization baseline for dense ViT prediction.