Search papers, labs, and topics across Lattice.
This study introduces the Lightweight Universal Mask Adapter (LUMA), a novel backbone-agnostic mask-transformer head that enables effective image segmentation by leveraging any transformer backbone as a black-box feature extractor through cross-attention. LUMA achieves accuracy comparable to the state-of-the-art efficient ViT-segmenter while significantly reducing computational costs, making it versatile across various backbone architectures. Benchmarking 20 backbones and 11 pretraining schemes on standard datasets reveals that the choice of pretraining objective is more critical than the architecture itself for segmentation performance.
LUMA reveals that the choice of pretraining objective trumps architectural complexity in driving segmentation quality, challenging existing assumptions in the field.
Comparing transformer backbones for image segmentation is confounded: each is paired with a different decoder, recipe, and pretraining, so reported differences rarely reflect the backbone itself. We introduce the Lightweight Universal Mask Adapter (LUMA), a lightweight, backbone-agnostic mask-transformer head that treats any backbone as a black-box feature extractor, letting a set of queries read from its features through cheap cross-attention. LUMA matches the accuracy of EoMT, the state-of-the-art efficient ViT-segmenter, at lower cost, while attaching unchanged to isotropic, hierarchical, convolutional, and mixture-of-experts backbones alike. Holding this head fixed, we benchmark 20 backbones, 11 pretraining schemes and a range of resolutions on ADE20K and Cityscapes under one modern recipe. We find that ``efficient''token mixers fail to deliver efficiency even at the high resolutions that motivate them, with plain ViT holding the throughput Pareto-front at every resolution. Additionally, the pretraining objective, not the architecture, the lever the field has tuned hardest, governs segmentation quality.
