Search papers, labs, and topics across Lattice.
This paper introduces DeepFusionKernel, a deeply fused kernel designed to optimize the memory bandwidth bottleneck caused by large SwiGLU MLP blocks in agentic LLM inference with long contexts. By reducing HBM traffic and improving cache reuse, DeepFusionKernel significantly accelerates inference. Experiments demonstrate speedups of up to 13.2% on H100 and 9.7% on A100 GPUs compared to SGLang.
Fusing kernels in SwiGLU MLP blocks slashes memory bandwidth bottlenecks, yielding up to 13.2% speedups on H100 GPUs during agentic LLM inference.
Agentic LLM inference with long contexts is increasingly limited by memory bandwidth rather than compute. In this setting, SwiGLU MLP blocks, whose large weights exceed cache capacity, become a major yet under-optimized bottleneck. We propose DeepFusionKernel, a deeply fused kernel that cuts HBM traffic and boosts cache reuse, delivering up to 13.2% speedup on H100 and 9.7% on A100 over SGLang. Integrated with SGLang and paired with a kernel scheduler, DeepFusionKernel ensures consistent accelerations over generation lengths, while remaining adaptable to diverse models, inference configurations, and hardware platforms.
