Search papers, labs, and topics across Lattice.
This paper introduces 3DLS, a novel 3D-stacked chiplet architecture designed to optimize large language model (LLM) serving by addressing the contention issues in conventional 2D/2.5D architectures. By utilizing vertical interconnects for KV-cache transfers while maintaining lateral D2D fabric for decode-side tensor parallelism, 3DLS significantly reduces communication latency and improves throughput. The results demonstrate a remarkable increase in performance, achieving up to 1.49 times higher throughput and 60.2% lower end-to-end latency compared to existing planar architectures.
Physical isolation in chiplet design can boost LLM serving performance by nearly 50% while slashing latency by over 60%.
Large language model (LLM) serving increasingly combines prefill-decode (PD) disaggregation with tensor parallelism (TP) to support large models and long contexts. In conventional 2D/2.5D chiplet architectures, layer-wise prefill-to-decode KV-cache transfer decode-side TP collectives share the same lateral die-to-die (D2D) interconnect, creating mixed-traffic contention on the decode critical path. This contention increases communication latency, prolongs token generation intervals, and degrades end-to-end serving performance. We propose 3DLS, a logic-on-logic 3D-stacked chiplet architecture that separates traffic classes by routing KV-cache transfers through vertical interconnects while preserving decode-side TP collectives on the lateral D2D fabric. 3DLS achieves up to 1.49$\times$ throughput and 60.2\% lower end-to-end (E2E) latency over the shared-fabric planar baseline, and still achieves up to 1.17$\times$ throughput and 31.4\% lower E2E latency over a workload-aware priority-managed planar baseline. These results highlight that physical isolation is an important design principle for future chiplet-based PD-disaggregated LLM serving systems.
