Search papers, labs, and topics across Lattice.
This paper introduces a hardware acceleration framework for block-diffusion large language models (LLMs) tailored for edge devices, addressing the inefficiencies of single-stream edge inference and full-attention recomputation. By co-designing a wide-I/O LPDDR system and innovative caching mechanisms, the authors achieve significant energy reductions and latency improvements for 1.5B and 7B model evaluations on Jetson-class platforms. The results show energy reductions of up to 3.96x and latency speedups of up to 4.44x, with minimal impact on model performance, highlighting the potential for efficient LLM deployment in resource-constrained environments.
Achieving nearly 4x energy savings and over 4x latency improvements for LLMs on edge devices without sacrificing performance opens new avenues for deploying advanced AI in real-world applications.
Single-stream (batch-one) edge inference cannot amortize weight traffic across requests. Full-attention diffusion LLMs recompute the entire sequence at every step; native block diffusion makes completed blocks immutable and exactly cacheable, yet refinement still streams prefix KV and FFN weights. We co-design WIFiV-LPDDR, a wide-I/O LPDDR system for precision-tagged reads, BRQ-KV for a canonical low-rank-plus-INT8-residual prefix with query-dependent per-entry precision, and DAT-FFN for drift-mapped canonical replacement, adjacent-stage-corrected low-bit delta, or cached-state carry while keeping live activations unquantized. Both map to an input-stationary mixed-precision systolic array. For the evaluated 1.5B/7B models on modeled Jetson-class platforms, the full stack provides arithmetic-mean energy-reduction factors of 3.79x/3.96x and arithmetic-mean latency speedups of 2.88x/4.44x at the reported DAT-FFN settings; every corresponding compressed model-benchmark score drops by less than one absolute percentage point from its baseline.
