Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces Triadic Suffix Tokenization (TST), a novel tokenization scheme designed to improve numerical reasoning in LLMs by consistently representing number magnitude. TST partitions numbers into three-digit triads and annotates each triad with explicit magnitude markers, creating a fixed mapping between suffixes and orders of magnitude. This approach aims to provide a consistent gradient signal during training, addressing the issue of inconsistent number fragmentation in standard subword tokenization methods.
LLMs can now represent numbers with explicit magnitude, potentially unlocking more reliable arithmetic and scientific reasoning.
Standard subword tokenization methods fragment numbers inconsistently, causing large language models (LLMs) to lose positional and decimal structure - a primary driver of errors in arithmetic and scientific reasoning. We introduce Triadic Suffix Tokenization (TST), a deterministic scheme that partitions digits into three-digit triads and annotates each triad with an explicit magnitude marker. Critically, the scheme defines a fixed, one-to-one mapping between suffixes and orders of magnitude for the integer part (thousands, millions, billions, etc.) and a parallel system of replicated markers for fractional depth (tenths, thousandths, millionths, etc.). Unlike approaches that rely on positional inference, this method provides a consistent gradient signal, which should ensure stable convergence. Two implementation variants are proposed: (1) a vocabulary-based approach that adds at most 10,000 fixed tokens to an existing vocabulary, covering 33 orders of magnitude ($10^{-15}$ to $10^{18}$); and (2) a suffix-marker approach that uses a small set of special tokens to denote magnitude dynamically. Both variants preserve exact digits while making order-of-magnitude relationships transparent at the token level. The framework is inherently scalable, allowing for linear vocabulary expansion to accommodate arbitrary precision and range. TST is architecture-agnostic and can be integrated as a drop-in preprocessing step. Experimental validation is deferred to future work.