Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
Addressing the quadratic scaling bottleneck of multi-view 3D vision transformers, the authors demonstrate that standard attention-mass pruning causes performance collapse because attention probability correlates poorly with functional output shifts. To fix this, they formulate ReSS (Residual-Restoring Sparse Attention), which reframes block-sparse attention as a problem of minimizing the directional drift imparted on the residual stream. Across three backbones and five datasets, ReSS consistently outperforms prior sparse attention methods, revealing that performance across disparate pruning techniques collapses onto a single universal curve when plotted against realized residual drift.
Attention probability is a deceptive metric for transformer sparsification—model degradation across multi-view vision transformers is universally governed by directional drift in the residual stream rather than lost attention mass.
3D vision transformers such as VGGT predict camera poses and scene geometry from multi-view images in a single forward pass, but their global attention over all concatenated view tokens dominates computation as the number of views grows. To reduce this cost, SparseVGGT and HeSS sparsify attention at the block level, and both retain blocks with high attention probability. However, we observe that attention probability poorly predicts how much the model's behavior actually changes when a block is removed, and we show that this mismatch is why performance collapses as sparsity increases. In this paper, we propose ReSS (ReSidual-ReStoring Sparse Attention), which recasts block selection from a problem of maximizing the retained attention mass to one of minimizing the drift that sparsification leaves in the residual stream. We introduce a drift score that quantifies how much each block shifts the residual, and, since the drift of a drop set depends on the directions of the contribution vectors rather than on their magnitudes alone, an iterative residual restoration procedure that refines the drop set as a whole. Across three backbones and five datasets, ReSS preserves dense performance better than prior methods at matched sparsity. Two further results support drift as the quantity that governs the cost of sparsification: maximizing drift degrades performance faster than random selection, and plotted against realized drift instead of sparsity, all methods fall approximately onto a single curve. Code is available at https://github.com/libary753/ReSS.