Search papers, labs, and topics across Lattice.
This paper introduces the state-prediction separation hypothesis, which posits that separating the roles of state storage and next-token prediction in Transformers can enhance language modeling performance. By designing a Transformer variant that employs two distinct computation streams for these functions, the authors conducted pretraining experiments that revealed significant improvements in validation loss and downstream task performance. The results indicate that this separation approach not only boosts efficiency but also consistently outperforms traditional Transformer architectures by 2-3 percentage points on average.
Separating state storage from next-token prediction in Transformers leads to a 2-3 percentage point improvement in downstream task performance.
Transformers use the same forward computation stream to both predict the next token and store useful state for future token predictions. We formulate the \emph{state-prediction separation hypothesis}: disentangling the two roles yields better language modeling performance. We design a Transformer variant that uses two computation streams to separate the two functions, and conduct pretraining experiments across various scales. Our experiments show that state-prediction separation consistently offers better data and compute efficiencies, improving validation loss and outperforming standard Transformers by 2--3 percentage points on average on downstream tasks. We also conduct extensive empirical analysis that rules out potential confounders and demonstrates the fundamental difference in the gradients our design entails.
