Pier-Jean Malandrino

Sub-3-bit LLM inference finally sheds its dequantization bottleneck: decoding Leech lattices on-the-fly via a 16 KiB Golay trellis yields over 113 tokens per second while remaining within 2.5 MMLU points of 4-bit AWQ.

Achieving 2.27x faster decoding for 2-bit LLM weights could redefine efficiency benchmarks in low-bit quantization methods.