Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the limitations of W4A4 quantization in LLM inference due to the dequantization overhead on CUDA Cores, identifying the Tensor Cores to CUDA Cores throughput ratio ($ρ$) as a critical factor influencing performance. Through systematic benchmarking across various GPU architectures, the authors demonstrate that W4A4's effectiveness is highly platform-dependent, achieving significant speedups on certain models while underperforming on others. The proposed APEX4 framework optimizes pure INT4 GEMM kernels with $ρ$-aware granularity adaptation, resulting in notable improvements in speed and accuracy across multiple GPUs, including a 1.66x speedup on L40S and enhanced zero-shot accuracy on LLaMA-2-70B.
W4A4 quantization can yield up to 2.5x speedup on certain GPUs, but its effectiveness hinges on the hardware's compute balance.
W4A4 quantization promises full utilization of INT4 Tensor Cores, yet group dequantization overhead on CUDA Cores has driven existing systems to mixed-precision fallbacks. We present the first systematic study of how intra-SM compute balance governs this bottleneck. Through controlled benchmarks across four GPUs from Ampere and Ada architectures, we identify the Tensor Cores to CUDA Cores throughput ratio ($ρ$) as the primary hardware indicator: the W4A4-g128 kernel yields $2.0$--$2.5\times$ speedup on RTX~3090 ($ρ=16$) yet degrades to $0.43$--$0.47\times$ on A100 ($ρ=64$) in compute-bond scenarios, establishing W4A4 viability as platform-dependent rather than universally infeasible. Guided by this finding, we build \textbf{APEX4}, which co-designs pure INT4 GEMM kernels with $ρ$-aware granularity adaptation to mitigate the CUDA Cores dequantization bottleneck. APEX4 achieves perplexity within 0.63 of FP16 on LLaMA-2-70B and outperforms W4Ax Atom-g128 by 4.0\%--4.4\% in zero-shot accuracy. Deployed as a drop-in replacement in unmodified vLLM, it delivers up to $1.66\times$ end-to-end speedup on L40S ($ρ=8$), and $1.78\times$ on RTX~3090 ($ρ=16$), $2.09\times$ on A40 ($ρ=16$), while recovering A100 ($ρ=64$) to $1.20$--$1.40\times$ via the mixed-granularity mode.