Search papers, labs, and topics across Lattice.
This study systematically evaluates various attention-based memory constraints in transformer models to better align their performance with human reading times. By comparing static and dynamic memory constraints across different model sizes and training datasets, the authors find that content-sensitive constraints significantly outperform distance-based ones in predicting human behavior. Notably, the research reveals a trade-off between psychometric fit and grammatical competence, indicating that transformers may not universally model cognitive processes.
Content-sensitive attention constraints in transformers can dramatically enhance alignment with human reading times, revealing critical limitations in existing models.
Transformer-based language models are widely used as models of human language processing, yet their attention mechanisms allow lossless access to the full preceding context, unlike the limited memory systems of humans. We hypothesize that installing memory constraints into transformers'attention mechanisms can improve their fit to human behavioral data. While previous work has explored individual constraints in isolation, we conduct a systematic comparison of multiple attention-based memory mechanisms across different model sizes and training corpora, evaluating both psychometric predictive power for human reading times and grammatical competence. We additionally compare static constraints, in which the constraint strength is fixed throughout training, to dynamic memory curricula. We find that constraints that are sensitive to the content of intervening tokens consistently achieve the highest alignment with human reading times, outperforming distance-based constraints. We observe a dissociation between psychometric fit and grammatical competence under dynamic memory curricula, suggesting that Transformers cannot serve as a one-size-fits-all cognitive model.
