Search papers, labs, and topics across Lattice.
This paper identifies a critical boundary-minority underestimation failure in Vision Transformers, where small, high-response regions contribute significantly to Gibbs mass but are overlooked by block mean summarization. The authors formalize this issue through a discrepancy analysis between normalized log-mean-exp free energy and mean summarization, leading to the development of Boundary-Minority Free-Energy Adaptive Screening (BMFA). BMFA effectively reduces synthetic underestimation and improves performance metrics across multiple datasets, achieving significant reductions in mean gaps while maintaining high accuracy levels.
Boundary-Minority Free-Energy Adaptive Screening (BMFA) slashes underestimation errors in Vision Transformers, revealing hidden contributions that traditional methods miss.
Vision Transformers process spatially redundant tokens efficiently only when coarse token summaries preserve the evidence required by exponential attention aggregation. We identify a boundary-minority underestimation failure in which a spatially small, high-response region contributes dominant Gibbs mass while remaining nearly invisible to a block mean. We formalize the failure through the discrepancy between normalized log-mean-exp free energy and mean summarization, prove that minority Gibbs mass can remain non-vanishing as its spatial support and mean contribution vanish, and characterize the limitations of finite-order moment corrections. Building on the resulting analysis, we introduce Boundary-Minority Free-Energy Adaptive Screening (BMFA), which constructs a hierarchical piecewise-constant approximation and recursively refines blocks according to a computable lower-bound increment of local free energy. Controlled synthetic tests, COCO and LVIS diagnostic probes, closed-loop DeiT-Tiny evaluations, and ImageNet-1K experiments establish a consistent evidence chain. BMFA reduces the mean synthetic underestimate from 2.582 to 0.261 at a 5.794% leaf ratio, lowers the COCO image-edge mean gap from 2.254 to 0.526, and preserves 71.520% ImageNet Top-1 accuracy at a 55.861% leaf ratio. The current prototype evaluates selection quality after full QK computation; the reported leaf ratio therefore characterizes representation granularity rather than verified sparse-kernel speedup.
