Search papers, labs, and topics across Lattice.
This paper introduces DANMP, a near-memory processing (NMP) accelerator designed to address the memory access inefficiencies of Multi-Scale Deformable Attention (MSDAttn). DANMP employs non-uniform NMP integration, allocating processing elements (PEs) only to banks with frequently accessed data, and introduces a clustering and packing (CAP) method to improve temporal locality. Experimental results demonstrate that DANMP achieves significant speedup (97.43x) and energy efficiency (208.47x) improvements compared to an NVIDIA A6000 GPU in object detection inference.
Achieve 97x speedup and 208x energy efficiency for Multi-Scale Deformable Attention by co-designing a near-memory processing architecture that overcomes the limitations of existing NMP solutions.
Multi Scale Deformable Attention (MSDAttn) has become a fundamental component in various vision tasks due to its effective multi scale grid sampling (MSGS). However, its reliance on random sampling results in highly irregular memory access patterns, making it a memory intensive operation inefficient for GPUs. Near memory processing (NMP) offers a promising solution for accelerating memory bound kernels, yet existing NMP based attention accelerators remain suboptimal for MSDAttn due to incompatible load balancing and data reuse strategies. Specifically, current NMP solutions uniformly distribute processing elements (PEs) across all banks, leading to significant PE underutilization and excessive cross bank data transfers. Moreover, most rely on locality based reuse, which fails under MSDAttn's unpredictable sampling patterns. To address these challenges, this paper presents DANMP, a hardware software co designed NMP based MSDAttn accelerator. On the hardware side, DANMP adopts non uniform NMP integration to handle unbalanced workloads, allocating PEs only in select banks for hot entries, while cold data are processed at the bank group level reducing PE idleness and cross bank transfers. On the software side, it introduces a clustering and packing (CAP) method that leverages clustering to improve temporal locality in query processing, enhancing data reuse. Finally, we implement host NMP co optimization techniques, including an optimized programming model, customized instructions, and a tailored dataflow. Experiments on object detection inference show that DANMP achieves 97.43x speedup and 208.47x energy efficiency improvement over NVIDIA A6000 GPU.
