Search papers, labs, and topics across Lattice.
This paper introduces Superpixel Transformers (SPT), a framework unifying superpixel-based image classification with Vision Transformers (ViTs) by generalizing SICGAT and ViT architectures. SPT supports flexible superpixel chunking, connectivity graphs, and positional encodings, incorporating superpixel shape and color information. Experiments on CIFAR10, FashionMNIST, and Imagenette demonstrate SPT's superior performance over previous superpixel-based GNNs and competitive results with ViTs, highlighting the benefits of constrained graph connectivity for ViT performance.
Superpixel Transformers show that strategically chunking images into superpixels can boost performance over standard Vision Transformers, suggesting a new path for efficient and effective attention mechanisms.
Superpixel-based image classification has traditionally leveraged graph neural networks (GNNs) for processing irregular image representations. Recent advances in computer vision, driven by Vision Transformers (ViTs), have introduced new paradigms in self-attentional models, surpassing convolutional neural networks (CNNs) in various tasks. However, a synergistic connection between GNNs, superpixels, and transformers remains unexplored. In this work, we propose Superpixel Transformers (SPT), a novel framework that unifies superpixel-based image classification and ViTs. SPT generalizes the Superpixel Image Classification with Graph Attention Networks (SICGAT) model and ViT to support arbitrary superpixel-based chunking strategies, connectivity graphs, and positional encodings. We introduce refinements including a multidimensional sine-cosine positional encoding and an enriched patch data structure that fully incorporates superpixel shape and color information. By testing SPT across datasets such as CIFAR10, FashionMNIST, and Imagenette, with various superpixel generation and graph connectivity strategies, we demonstrate that SPT achieves superior performance compared to previous superpixel-based GNN methods and remains competitive with ViTs. Notably, our approach addresses the limitations of SICGAT, such as information loss during pixel aggregation, and shows how constrained graph connectivity can enhance ViT performance. SPT bridges the gap between superpixel-based and transformer models, opening avenues for cross-domain generalization and future innovations in hybrid attentional frameworks, and showing that an image can also be worth $16\times16$ superpixels.
