Search papers, labs, and topics across Lattice.
The paper introduces ExTernD, a novel post-training factorization method for large language model (LLM) weight matrices that achieves high accuracy in ternary quantization by expanding the inner rank beyond full rank. This approach allows for the correction of quantization errors through additional components, enabling the method to reach bf16 accuracy levels while maintaining flexibility in memory and compute requirements. The results demonstrate that ExTernD can match or exceed the accuracy of existing quantization schemes, achieving a perplexity of 10.10 on wikitext-2, outperforming bf16 and positioning itself within the accuracy range of advanced quantization methods like Q4_K and Q5_K.
ExTernD achieves near-bf16 accuracy in ternary quantization, outperforming traditional methods by correcting quantization errors with expanded rank components.
We introduce ExTernD (Expanded-rank Ternary Decomposition), a post-training factorization of each LLM weight matrix $A \in \mathbb{R}^{m \times n}$ into $A \approx B \mathrm{diag}(D) C$ with ternary factors $B \in \{-1,0,+1\}^{m \times k}$, $C \in \{-1,0,+1\}^{k \times n}$ and a real scale vector $D \in \mathbb{R}^k$. The inner rank $k = \mu \min(m,n)$ is deliberately expanded beyond full rank ($\mu>1$), so that components past full rank correct the quantization error of earlier ones. We prove the residual decreases monotonically in $k$ and can be driven below any $\varepsilon>0$: ExTernD approaches bf16 accuracy arbitrarily closely, which no ternary scheme with a fixed plane count can do. Memory and compute scale continuously with $\mu$, and factor sparsity continuously with a threshold $\tau$, so an accuracy target is hit exactly rather than rounded to the next bit-width. ExTernD matches Q4_K's per-matrix accuracy at 5.2-5.5 effective bpw (5.1-5.5 with importance weighting) on Gemma-4-E2B and Qwen3.5-4B, and a full Qwen3.5-4B conversion at $\mu = 3$ reaches 10.10 wikitext-2 perplexity against 9.78 for bf16 (+3.2%), placing it near the Q4_K/Q5_K accuracy band at ~5.7 effective bpw.
