Search papers, labs, and topics across Lattice.
The paper introduces Adaptive Token Dictionary (ATD), a novel transformer architecture for image restoration that achieves global dependency modeling with linear complexity. ATD uses a learnable token dictionary to summarize image priors and a token dictionary cross-attention (TDCA) mechanism to enhance input features. By grouping input features into categories based on TDCA attention maps and integrating this information into the feed-forward network, ATD achieves state-of-the-art performance on image super-resolution, denoising, and JPEG artifact removal benchmarks.
Achieve state-of-the-art image restoration results with a novel transformer architecture that scales linearly with image size, overcoming the quadratic complexity bottleneck of self-attention.
Recently, Transformers have gained significant popularity in image restoration tasks such as image super-resolution and denoising, owing to their superior performance. However, balancing performance and computational burden remains a long-standing problem for transformer-based architectures. Due to the quadratic complexity of self-attention, existing methods often restrict attention to local windows, resulting in limited receptive field and suboptimal performance. To address this issue, we propose Adaptive Token Dictionary (ATD), a novel transformer-based architecture for image restoration that enables global dependency modeling with linear complexity relative to image size. The ATD model incorporates a learnable token dictionary, which summarizes external image priors (i.e., typical image structures) during the training process. To utilize this information, we introduce a token dictionary cross-attention (TDCA) mechanism that enhances the input features via interaction with the learned dictionary. Furthermore, we exploit the category information embedded in the TDCA attention maps to group input features into multiple categories, each representing a cluster of similar features across the image and serving as an attention group. We also integrate the learned category information into the feed-forward network to further improve feature fusion. ATD and its lightweight version ATD-light, achieve state-of-the-art performance on multiple image super-resolution benchmarks. Moreover, we develop ATD-U, a multi-scale variant of ATD, to address other image restoration tasks, including image denoising and JPEG compression artifacts removal. Extensive experiments demonstrate the superiority of out proposed models, both quantitatively and qualitatively.
