Search papers, labs, and topics across Lattice.
The Clustered Unit-level Similarity Transformer (CUST) addresses the inefficiencies of Vision Transformer (ViT)-based models in image super-resolution by integrating global and local information through a novel architecture. By allowing each patch to attend to similar patches beyond its local window and employing overlapping attention windows, CUST effectively captures long-range dependencies while maintaining computational efficiency. Experimental results show that CUST achieves a significant reduction in memory usage and inference time without sacrificing restoration quality compared to existing models.
CUST slashes memory usage and boosts inference speed while enhancing image super-resolution performance by intelligently aggregating patch similarities.
Recently, Vision Transformer (ViT)-based models have exhibited remarkable performance in image super-resolution. However, the quadratic computational complexity of ViTs with respect to spatial resolution severely constrains their efficiency, leading to high latency and massive memory consumption. To alleviate this, various window-based attention mechanisms have been proposed; yet, they inherently compromise the long-range dependency modeling that is the primary advantage of ViTs. To overcome these limitations, we propose the Clustered Unit-level Similarity Transformer (CUST), a novel architecture that efficiently integrates global and local information. Specifically, CUST enables each patch to aggregate and attend to similar patches within a broadened regional scope outside its local window, thereby capturing extensive contextual understanding. Furthermore, it employs overlapping attention windows to capture local dependencies, while explicitly extracting high-frequency details by computing the residual difference between the original features and their downsampled-upsampled counterparts. Comprehensive experiments demonstrate that our proposed model achieves a practical balance between computational efficiency and restoration performance. It achieves a lower memory footprint and faster inference speed compared to recent global context or lightweight models under realistic constraints. Code is available at [https://github.com/jwgdmkj/CUST].
