Search papers, labs, and topics across Lattice.
AdaFuse addresses the inference latency bottleneck of dynamic adapter integration in LLMs, which arises from fragmented CUDA kernel launches. They introduce a token-level pre-gating strategy that makes a single routing decision for all adapter layers, effectively staticizing the execution path per token. A custom CUDA kernel then performs a fused switching operation, merging selected LoRA adapter parameters into the backbone model in a single pass, achieving a 2.4x speedup.
Decoding LLMs with dynamic adapters doesn't have to be 2.5x slower: AdaFuse slashes latency by 2.4x with token-level pre-gating and fused kernel optimization.
The integration of dynamic, sparse structures like Mixture-of-Experts (MoE) with parameter-efficient adapters (e.g., LoRA) is a powerful technique for enhancing Large Language Models (LLMs). However, this architectural enhancement comes at a steep cost: despite minimal increases in computational load, the inference latency often skyrockets, leading to decoding speeds slowing by over 2.5 times. Through a fine-grained performance analysis, we pinpoint the primary bottleneck not in the computation itself, but in the severe overhead from fragmented, sequential CUDA kernel launches required for conventional dynamic routing. To address this challenge, we introduce AdaFuse, a framework built on a tight co-design between the algorithm and the underlying hardware system to enable efficient dynamic adapter execution. Departing from conventional layer-wise or block-wise routing, AdaFuse employs a token-level pre-gating strategy, which makes a single, global routing decision for all adapter layers before a token is processed. This"decide-once, apply-everywhere"approach effectively staticizes the execution path for each token, creating an opportunity for holistic optimization. We capitalize on this by developing a custom CUDA kernel that performs a fused switching operation, merging the parameters of all selected LoRA adapters into the backbone model in a single, efficient pass. Experimental results on popular open-source LLMs show that AdaFuse achieves accuracy on par with state-of-the-art dynamic adapters while drastically cutting decoding latency by a factor of over 2.4x, thereby bridging the gap between model capability and inference efficiency.
