Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the critical issue of prefix invariance violation in low-precision fast matrix multiplication (MatMul) for large language models, where rounding errors in mixed-token sums can corrupt likelihood-based scoring. The authors demonstrate that even FP8 MatMul realizations achieving high accuracy can alter model outputs on downstream tasks like OpenBookQA, highlighting the inadequacy of accuracy as a sole metric for prefix invariance. They introduce a novel approach using bounded integer codes and independent row quantization to construct certified MatMul realizations that guarantee bitwise equality to a prescribed row-local int8 operator, thus preserving prefix invariance and making realization choice a cost-optimization problem.
Low-precision matrix multiplication, even when repaired for accuracy, can break LLM prefix invariance and alter downstream task performance, necessitating a new certification approach.
Fast matrix multiplication saves multiplications through exact cancellation, but rounding sums that mix token rows can leave contributions from later tokens in earlier language model outputs. This threatens prefix invariance, which multiple-choice likelihood scoring relies on: a scored likelihood must depend only on its allowed prefix. On Qwen2.5-14B-Instruct, two fast FP8 realizations repaired to ordinary-looking accuracy still change the answers chosen by likelihood on 5.83% and 10.00% of 240 OpenBookQA items when only the text after the allowed prefix is replaced with the bf16 model's own greedy continuation. Both row-local controls, the bf16 model and a deployed FP8 matrix multiplication kernel, change none. Accuracy thus does not certify prefix invariance, and the stability criteria we analyze cannot tell realizations apart: across all 512 sign variants of two-level Strassen they stay constant while teacher-forced perplexities span a 772.4$\times$ range on the same model. We therefore construct certified realizations of two-level Strassen on bounded integer codes that quantize token rows independently, then mix and cancel exactly before rescaling, using 49 block multiplications instead of 64. Our certificate guarantees bitwise equality to a prescribed row-local classical int8 operator at the same quantization specification, so every certified realization inherits its prefix invariance. Certification thus turns realization choice into a pure cost decision: which certified realization runs can no longer change a single scored likelihood.