Search papers, labs, and topics across Lattice.
This paper re-evaluates the role of tokenization in language models, arguing that it should be viewed not just as an input preprocessing step but as a crucial determinant of output supervision. Through controlled experiments on numeric reasoning, the authors demonstrate that variations in output tokenization significantly influence task performance, training dynamics, and internal model representations, independent of input tokenization. This finding highlights that model comparisons may often reflect differences in task definitions rather than inherent model capabilities, a nuance overlooked in the majority of recent literature on numeric reasoning.
Tokenization isn鈥檛 just a preprocessing step; it鈥檚 a hidden driver of model performance and learning dynamics that could redefine how we interpret model evaluations.
Tokenization in language models is treated by default as an input preprocessing decision. We argue that this framing is incomplete: in autoregressive models, tokenizer granularity determines what the model must resolve in a single forward pass, and therefore the supervision signal it receives. This affects both the difficulty of the learning problem and the representations that emerge inside the model. We test this in a controlled experiment on numeric reasoning with a novel decoupling of input and output tokenization. As the output supervision view predicts, differences in task performance, training dynamics, and model internals are induced by output tokenization and largely invariant to input tokenization. This may matter in practice, because models with different tokenization strategies differ not only in input representation but in the task they were trained on. Comparisons between models may thus partly reflect task definition rather than ability. A survey of 120 recent *CL papers on numeric reasoning confirms that this is rarely acknowledged: only about 10% report the numeric tokenization of the models they evaluate, while 69% compare across tokenization, and thus supervision, regimes without reporting it. While prior work documents that tokenization consistently affects model performance, there is no principled account of why. We argue that framing tokenization as output supervision provides that account.
