Search papers, labs, and topics across Lattice.
This paper critiques the application of Surprisal Theory in computational psycholinguistics, arguing that the reliance on large language models (LLMs) obscures the necessary representational choices inherent in computational-level explanations. Through three analyses, the authors demonstrate that the algorithm and model architecture significantly influence language model probabilities, challenging the notion that LLM-surprisal can be treated as uniform across different systems. The findings call for a reevaluation of how researchers utilize LLM probabilities in testing Surprisal Theory, emphasizing the importance of recognizing underlying representational commitments.
Treating LLM-surprisal as interchangeable risks masking crucial representational choices that shape model behavior.
Surprisal Theory is often characterized as a computational-level explanation per (Marr, 1982). We argue in this work that, even though a computational level narrative has been used to support"representation-agnostic research"within computational psycholinguistics, the movement toward black box systems embodied by large language models (LLMs) does not exempt modelers using the surprisal metric from the representational decisions required by computational-level characterizations. In fact, we argue that the uncritical use of LLM-surprisal obfuscates the representational and algorithmic-level commitments of different models. In three analyses, we show that the choice of algorithm and model architecture play significant roles in the computation of language model probabilities. We advise that researchers who wish to test Surprisal Theory re-evaluate the practice of treating large language model probabilities as interchangeable
