Search papers, labs, and topics across Lattice.
HybridGen is introduced as a novel framework for efficient LLM inference that leverages both CPU and GPU resources for attention computation, particularly in systems with CXL-expanded memory. It addresses challenges like multi-dimensional attention dependencies, CPU-GPU load imbalance, and NUMA penalties through attention logit parallelism, a feedback-driven scheduler, and semantic-aware KV cache mapping. Experiments demonstrate that HybridGen achieves 1.41x-3.2x speedups compared to state-of-the-art KV cache management methods while maintaining accuracy across various LLMs and GPU platforms.
Squeeze up to 3.2x more performance from your long-context LLMs by intelligently splitting attention computation between CPU and GPU.
As modern LLMs support thousands to millions of tokens, KV caches grow to hundreds of gigabytes, stressing memory capacity and bandwidth. Existing solutions, such as KV cache pruning and offloading, alleviate these but underutilize hardware by relying solely on either GPU or CPU for attention computing, and considering yet limited CPU local memory for KV cache storage. We propose HybridGen, an efficient hybrid attention framework for long-context LLM inference. HybridGen enables CPU-GPU collaborative attention on systems with expanded tiered memory (e.g., CXL memory), addressing three key challenges: (1) multi-dimensional attention dependencies, (2) intensifying CPU-GPU load imbalance with longer sequences, and (3) NUMA penalty of tiered memories. HybridGen tackles these by introducing attention logit parallelism, a feedback-driven scheduler, and semantic-aware KV cache mapping. Experiments with three LLM models with eleven different sizes on three GPU platforms with a CXL-expanded memory show that HybridGen outperforms six state-of-the-art KV cache management methods by 1.41x--3.2x on average while maintaining superior accuracy.
NVIDIA