Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Blink is a novel LLM serving architecture that eliminates CPU involvement in the steady-state inference path by delegating request handling to a SmartNIC and utilizing a persistent GPU kernel for batching, scheduling, and KV-cache management. This is achieved by offloading request handling to the SmartNIC, which delivers inputs directly into GPU memory via RDMA. Evaluation shows Blink significantly outperforms existing systems like TensorRT-LLM, vLLM, and SGLang, especially under CPU interference, with improvements in TTFT, TPOT, decode throughput, and energy efficiency.
LLM serving bottlenecks on the CPU? Blink bypasses the CPU entirely, using SmartNICs and persistent GPU kernels to boost throughput by 2x and cut energy use nearly in half.
Large Language Model (LLM) inference is rapidly becoming a core datacenter service, yet current serving stacks keep the host CPU on the critical path for orchestration and token-level control. This makes LLM performance sensitive to CPU interference, undermining application colocation and forcing operators to reserve CPU headroom, leaving substantial capacity unutilized. We introduce Blink, an end-to-end serving architecture that removes the host CPU from the steady-state inference path by redistributing responsibilities across a SmartNIC and a GPU. Blink offloads request handling to the SmartNIC, which delivers inputs directly into GPU memory via RDMA, and replaces host-driven scheduling with a persistent GPU kernel that performs batching, scheduling, and KV-cache management without CPU involvement. Evaluated against TensorRT-LLM, vLLM, and SGLang, Blink outperforms all baselines even in isolation, reducing pre-saturation P99 TTFT by up to 8.47$\times$ and P99 TPOT by up to 3.40$\times$, improving decode throughput by up to 2.1$\times$, and reducing energy per token by up to 48.6$\%$. Under CPU interference, Blink maintains stable performance, while existing systems degrade by up to two orders of magnitude.