Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces LEXI, a lossless exponent compression scheme based on Huffman coding, to reduce data movement overheads in inter-chiplet communication for LLMs. LEXI compresses and decompresses BF16 exponents on the fly, leveraging the low Shannon entropy observed in exponent streams of LLM activations and caches. Results show that LEXI reduces inter-chiplet communication latency by 33-45% and end-to-end inference latency by 30-35% on Jamba, Zamba, and Qwen LLMs, with minimal area and energy overhead.
LLMs can run up to 35% faster on chiplet architectures thanks to a new lossless exponent compression technique that slashes inter-chiplet communication overhead.
Data movement overheads increase the inference latency of state-of-the-art large language models (LLMs). These models commonly use the bfloat16 (BF16) format for stable training. Floating-point standards allocate eight bits to the exponent, but our profiling reveals that exponent streams exhibit fewer than 3 bits Shannon entropy, indicating high inherent compressibility. To exploit this potential, we propose LEXI, a novel lossless exponent compression scheme based on Huffman coding. LEXI compresses activations and caches on the fly while storing compressed weights for just-in-time decompression near compute, without sacrificing system throughput and model accuracy. The codecs at the ingress and egress ports of network-on-chip routers sustain the maximum link bandwidth via multi-lane LUT decoders, incurring only 0.09 percent area and energy overheads with GF 22 nm technology. LEXI reduces inter-chiplet communication and end-to-end inference latencies by 33-45 percent and 30-35 percent on modern Jamba, Zamba, and Qwen LLMs implemented on a homogeneous chiplet architecture.