Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
DUET is a disaggregated accelerator architecture tailored for hybrid Mamba-Transformer LLMs, addressing the performance bottlenecks caused by the distinct compute requirements of prefill and decode phases. It employs a Prefill package with systolic arrays for efficient matrix multiplications and long-sequence SSMs, and a Decode package with vector-unit arrays for token-by-token SSM and vector-matrix multiplications. Evaluations on models like Nemotron-H-56B and Llama3-8B demonstrate that DUET achieves significant performance gains, including 4x faster time to first token and 1.4x higher throughput, compared to a B200 GPU.
Hybrid Mamba-Transformer models can get 4x faster time to first token and 1.4x higher throughput by disaggregating prefill and decode phases onto specialized accelerator packages.
Large language models operate in distinct compute-bound prefill followed by memory bandwidth-bound decode phases. Hybrid Mamba-Transformer models inherit this asymmetry while adding state space model (SSM) recurrences and element-wise operations that map poorly to matmul-centric accelerators. This mismatch causes performance bottlenecks, showing that a homogeneous architecture cannot satisfy all requirements. We introduce DUET, a disaggregated accelerator that assigns prefill and decode phases to specialized packages. The Prefill package utilizes systolic array chiplets with off-package memory for efficient large matrix multiplications and long-sequence SSMs. The Decode package utilizes vector-unit arrays with high-bandwidth in-package memory to accelerate token-by-token SSM and vector-matrix multiplications. Both architectures are runtime-configurable to support hybrid models with mixed Mamba and attention layers. Evaluations on Nemotron-H-56B, Zamba2-7B, and Llama3-8B across four workloads show that DUET achieves 4x faster time to first token, 1.4x higher throughput, and 1.5x lower time between tokens over the B200 GPU.