Search papers, labs, and topics across Lattice.
This paper explores the impact of working memory constraints on Transformer performance in low-data regimes by implementing fixed-width and temporal decay attention mechanisms within GPT-2. The modified models, trained on 10M and 100M word datasets, demonstrate that fixed-width attention significantly improves grammatical accuracy (BLiMP) and alignment with human reading times compared to standard attention, particularly when data is scarce. This suggests that cognitively-inspired constraints can act as a beneficial inductive bias for learning robust linguistic representations.
Fixed-width attention spans can give you better grammar and human-like reading patterns, especially when you're short on training data.
We investigate the integration of human-like working memory constraints into the Transformer architecture and implement several cognitively inspired attention variants, including fixed-width windows based and temporal decay based attention mechanisms. Our modified GPT-2 models are trained from scratch on developmentally plausible datasets (10M and 100M words). Performance is evaluated on grammatical judgment tasks (BLiMP) and alignment with human reading time data. Our results indicate that these cognitively-inspired constraints, particularly fixed-width attention, can significantly improve grammatical accuracy especially when training data is scarce. These constrained models also tend to show a stronger alignment with human processing metrics. The findings suggest that such constraints may serve as a beneficial inductive bias, guiding models towards more robust linguistic representations, especially in data-limited settings.
