Search papers, labs, and topics across Lattice.
This paper optimizes CUDA kernels for depthwise convolution, a key operation in S4ConvD, by systematically exploring naive, coalesced, cache-blocked, and warp-tiled implementations. They introduce a counter-free performance analysis methodology combining CUDA-event timing, memory-traffic modeling, and roofline analysis to gain architectural insights in cloud environments without hardware counters. Results show a 3.26x speedup in convolution runtime using the warp-tiled kernel and a 1.29x end-to-end training speedup, highlighting the importance of memory access efficiency and on-chip data reuse.
Unlock significant speedups in depthwise convolutions (up to 3.26x) with optimized CUDA kernels, even in restricted cloud environments lacking hardware performance counters.
Efficient GPU execution of convolution operators is governed by memory-access efficiency, on-chip data reuse, and execution mapping rather than arithmetic throughput alone. This paper presents a controlled operator-level study of CUDA kernel optimization for the depthwise convolution used in Structured State Space Model Convolutional Diagonal (S4ConvD), together with a cloud-compatible, counter-free performance analysis methodology. The operator, model, dataset, and training configuration are fixed, and only the CUDA kernel implementation is varied. The evaluated CUDA kernels comprise naive, global-memory-coalesced, shared-memory cache-blocked, and warp-tiled variants, covering forward, input-gradient, and weight-gradient execution paths under steady-state training conditions. Performance is characterized using a counter-free methodology that combines CUDA-event timing, execution-path decomposition, analytically derived memory-traffic modeling, effective-bandwidth estimation, and roofline analysis. This enables profiling-like architectural insights without requiring hardware performance counters or privileged profiling access. The warp-tiled kernel reduces convolution runtime by $3.26\times$ relative to the naive CUDA baseline, while end-to-end training speedup reaches $1.29\times$. A PyTorch implementation is used separately for numerical validation and runtime context, but is not treated as a controlled architectural baseline. Forward and input-gradient paths benefit substantially from improved locality and on-chip data reuse, whereas the reduction-dominated weight-gradient path remains the primary bottleneck. The results demonstrate that meaningful architecture-level GPU kernel analysis can be performed reproducibly in restricted cloud environments, even without access to hardware performance counters.
