Search papers, labs, and topics across Lattice.
This paper introduces a hardware-software co-design framework aimed at optimizing N:M sparse Vision Transformer (ViT) inference to address the significant latency issues associated with existing models. By developing an N:M sparse CUDA kernel (MD-SpMM) and implementing a layer-wise sparsity search that adheres to end-to-end latency budgets, the authors achieve over 2.2x speedup in inference time without sacrificing accuracy. The findings highlight the importance of coordinated design between software and hardware to effectively leverage sparsity for practical acceleration in deep learning applications.
Achieving over 2.2x inference speedup in Vision Transformers while maintaining accuracy reveals the untapped potential of hardware-software co-design in optimizing model performance.
Vision Transformers (ViTs) achieve strong accuracy but incur high inference latency. Semi-structured N:M sparsity can reduce arithmetic cost, yet its theoretical savings often fail to translate into proportional end-to-end speedups on modern GPUs. This mismatch arises because deployment latency depends not only on arithmetic reduction but also on execution regularity and hardware scheduling under sparsity. Achieving practical acceleration, therefore, requires coordinated design across sparse execution and sparsity configuration. To this end, we propose a hardware-software co-design framework for N:M sparse ViT inference. On the hardware side, we design MD-SpMM, an N:M sparse CUDA kernel that reorganizes sparse GEMM into micro-dense, Tensor-Core-aligned dataflow and uses inference-aware adaptive parallelism to sustain utilization. On the software side, we perform layer-wise sparsity search under explicit end-to-end latency budgets using a three-stage heuristic search with constraint relaxation to avoid premature convergence and enable deployment-aware sparsity allocation. Experiments on multiple ViT/Swin models and GPU platforms show that the framework achieves over 2.2x latency speedup while maintaining comparable accuracy and delivering superior accuracy under the same latency constraint. The source code is publicly available at https://github.com/liuganhuo/realizable-nm-sparse-transformer.
