Search papers, labs, and topics across Lattice.
This paper addresses the mismatch between linguistically motivated units used in psycholinguistic experiments and the fixed token alphabets used by pretrained language models when calculating surprisal. They propose a unified framework for reasoning about surprisal over arbitrary unit inventories, explicitly disentangling the definition of the unit of analysis from the choice of regions of interest. The authors argue that tokenization should be treated as an implementation detail, improving the validity of surprisal-based analyses.
Surprisal theory's reliance on arbitrary tokenization schemes undermines its validity, but this framework offers a way to fix it.
Surprisal theory links human processing effort to the predictability of an upcoming linguistic unit, but empirical work often leaves the notion of a unit underspecified. In practice, experimental stimuli are segmented into linguistically motivated units (e.g., words), while pretrained language models assign probability mass to a fixed token alphabet that typically does not align with those units. As a result, surprisal-based predictors depend implicitly on ad hoc procedures that conflate two distinct modeling choices: the definition of the unit of analysis and the choice of regions of interest over which predictions are evaluated. In this paper, we disentangle these choices and give a unified framework for reasoning about surprisal over arbitrary unit inventories. We argue that surprisal-based analyses should make these choices explicit and treat tokenization as an implementation detail rather than a scientific primitive.
