Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates energy-based transformers as predictors of reading difficulty, linking them to associative memory models and contrasting them with traditional measures like surprisal and attention entropy. The authors demonstrate that energy measures significantly improve the prediction of reading times across various corpora, outperforming surprisal alone. Notably, in a controlled experiment, energy at a single layer effectively captures the object/subject asymmetry in relative clause processing, indicating its potential as a unified predictor in computational psycholinguistics.
Energy-based transformers can predict reading difficulty more effectively than traditional measures, unifying multiple predictors into a single robust framework.
Transformer language models have become established tools for modeling human sentence processing, with measures such as surprisal and attention entropy serving as effective predictors of reading difficulty that together capture complementary aspects of processing load. Here, we explore a related class of transformer models: energy-based transformers, which provide a principled formal link to associative memory models, bringing processing research into direct contact with the broader literature on Hopfield networks and dense associative memory. To our knowledge, this is the first exploration of an energy-based transformer measure in computational psycholinguistics. Across reading-time corpora (Natural Stories, UCL eye-tracking, UCL self-paced reading), the energy measure is a robust predictor of reading times, providing significant fit beyond surprisal in all three. In a controlled experiment on relative clause processing, energy at a single layer captures the well-known object/subject asymmetry. We find evidence that it subsumes effects attributable to both attention entropy and surprisal, suggesting that energy may serve as a single unified predictor where multiple complementary measures have previously been required.