Search papers, labs, and topics across Lattice.
This paper analyzes the design space for heterogeneous prefill-decode (PD) inference in large language models (LLMs), focusing on the interactions between accelerator types, numerical precision, interconnects, and KV state management. By identifying key boundary decisions鈥攃ompute placement, KV representation, and KV ownership鈥攖he authors provide actionable insights into optimizing performance and resource allocation in LLM deployments. The findings highlight that precision policies should be runtime-specific rather than fixed, and that KV transfer mechanisms must consider representation compatibility to avoid bottlenecks in heterogeneous environments.
Only a subset of design interactions in heterogeneous LLM inference are binding constraints, revealing critical insights for optimizing deployment strategies.
Heterogeneous prefill-decode (PD) inference is now in production: prefill on cost-efficient or supply-available accelerators, decode on bandwidth-strong ones, and KV state crossing mixed interconnects in mixed numerical formats. Each deployment makes these decisions on its own. What is missing is the picture across configurations-which decisions must be made jointly at the PD boundary, and which can be made independently. We propose a design space organized along four design axes-accelerator, precision, interconnect, and KV residency and the workload regime (stage pressure) they respond to. We show that only a subset of interactions among these factors become binding constraints once PD inference becomes heterogeneous. These interactions surface through three recurring boundary decisions: compute placement, KV representation, and KV ownership. The resulting analysis yields concrete guidance. Precision policy belongs to runtime roles rather than to a single system-wide setting, because the same low-bit format relieves different bottlenecks on each side of the boundary. KV transfer engines move bytes rather than tensor semantics, making representation compatibility an explicit boundary concern whenever producer and consumer differ. The KV handoff also carries a lifecycle-reservation, release, and failure recovery-that spans prefill and decode and requires explicit ownership. Two further interactions remain open. Cross-vendor and interconnect-related claims are stated as design guidance grounded in industrial deployment observations and source-code inspection of the runtimes involved.
