Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To eliminate the massive lookup table and memory overheads that plague Leech-lattice quantization, Tetra implements a compact 64-state Golay trellis decoding scheme that compresses 24-weight blocks into 48 bits and reduces GPU memory reads to just 2.15 bits per weight. Sub-3-bit quantization has historically faced steep dequantization latency and severe accuracy drops, hindering its deployment in production inference. Evaluated on Qwen3 models at ~2.7 bits per parameter, Tetra achieves 113.8 tokens per second on an NVIDIA L40S, trails 4-bit AWQ by only 2.46 to 4.76 MMLU points despite halving the footprint, and beats llama.cpp's IQ2_XXS by 23.6 points at the 4B scale.
Sub-3-bit LLM inference finally sheds its dequantization bottleneck: decoding Leech lattices on-the-fly via a 16 KiB Golay trellis yields over 113 tokens per second while remaining within 2.5 MMLU points of 4-bit AWQ.
Leech-lattice quantization gives good quality at two bits per weight, but its codebooks hold more than 10^14 points, too many for a lookup table. Our earlier kernel expanded the codes at load time and read 4.804 bits per weight from GPU memory for 2 bits of code. We present Tetra, a new codebook on the same lattice. A 24-weight block still takes 48 bits, most of which index a 64-state trellis of the Golay code and one shared 16 KiB table. The kernel decodes a block with six table loads and two small lookups inside the matrix-vector product, and reads 2.148 bits per weight. For full models, we retrain one scale per matrix row, store the matrices that lose the most as 4-bit integers, and pay for them with 4-bit embedding tables. Our Qwen3-4B, 8B and 14B files hold 2.73, 2.70 and 2.73 bits per parameter over the whole model. They score 63.37, 69.58 and 75.66 on the full MMLU test set, 4.76, 4.21 and 2.46 points below 4-bit AWQ at 5.3 to 6.0 bits per parameter. They generate 113.8, 95.0 and 57.2 tokens per second in our engine. On GSM8K, through the served kernel, they lose 9.63, 4.62 and 3.26 points to FP16. At 4B our file scores 23.6 points above llama.cpp's IQ2_XXS (2.48 bits per parameter). Every number we measured for a table or figure comes from one NVIDIA L40S GPU. We preregistered the main experiments.