Search papers, labs, and topics across Lattice.
This paper investigates whether language model representations capture cognitive signals related to human reading times by probing various layers of LMs against human eye-tracking data across five languages. Regularized linear regression is used to predict early- and late-pass reading measures from LM representations and scalar predictors like surprisal. The key finding is that early layers of LMs outperform surprisal in predicting early-pass reading measures, suggesting alignment between model depth and human reading stages, while surprisal remains superior for late-pass measures.
Early layers of language models capture human-like processing signatures in reading, rivaling traditional measures like surprisal in predicting initial eye movements.
Probing has shown that language model representations encode rich linguistic information, but it remains unclear whether they also capture cognitive signals about human processing. In this work, we probe language model representations for human reading times. Using regularized linear regression on two eye-tracking corpora spanning five languages (English, Greek, Hebrew, Russian, and Turkish), we compare the representations from every model layer against scalar predictors -- surprisal, information value, and logit-lens surprisal. We find that the representations from early layers outperform surprisal in predicting early-pass measures such as first fixation and gaze duration. The concentration of predictive power in the early layers suggests that human-like processing signatures are captured by low-level structural or lexical representations, pointing to a functional alignment between model depth and the temporal stages of human reading. In contrast, for late-pass measures such as total reading time, scalar surprisal remains superior, despite its being a much more compressed representation. We also observe performance gains when using both surprisal and early-layer representations. Overall, we find that the best-performing predictor varies strongly depending on the language and eye-tracking measure.
ETH