Search papers, labs, and topics across Lattice.
This study reveals that the per-word processing time in the state-space language model Mamba aligns closely with human reading times, suggesting a dynamic relationship between model architecture and cognitive processing. By analyzing a naturalistic reading dataset, the authors demonstrate that Mamba's timestep is a significant predictor of human reading times, even when controlling for other factors like GPT-2 surprisal. The findings indicate that Mamba's architecture provides a novel framework for understanding human language processing, particularly in terms of memory dynamics and information retention across different time scales.
Mamba's per-word processing time aligns with human reading speeds, offering a new perspective on cognitive language processing.
This study demonstrates an alignment of per-word processing time in a popular state-space language model Mamba and human readers. In Mamba, the recurrent state transition at each layer conceptually takes some duration of time, the discretization timestep $螖_t$, determined dynamically in response to the input. Using a naturalistic reading dataset, we show that the per-word timestep from Mamba is a significant predictor of human reading times, and remains significant even when known predictors such as GPT-2 surprisal are controlled for. We further suggest, through formal analysis of Mamba's architecture and internal dynamics, that Mamba can serve as a new, valuable lens to look at human real-time language processing with ever-updated memory, because it allows us to look at how each module (layer) weighs short- and long-term information retention, and how noise may interact with dynamic, continuous memory representation. Code is available online.
