Search papers, labs, and topics across Lattice.
This paper investigates the parallelizability of linear RNNs (LRNNs) compared to traditional nonlinear RNNs and transformers. It establishes a connection between RNN types and complexity classes, showing that LRNNs can be viewed as log-depth arithmetic circuits, while nonlinear RNNs can solve $\mathsf{L}$-complete and $\mathsf{P}$-complete problems. The analysis also differentiates the expressivity of LRNN variants, linking them to automata-theoretic models and revealing tradeoffs between expressivity and parallelism.
Linear RNNs achieve transformer-like parallelization because they're essentially log-depth arithmetic circuits, while nonlinear RNNs are fundamentally limited by their ability to solve computationally harder problems.
The community is increasingly exploring linear RNNs (LRNNs) as language models, motivated by their expressive power and parallelizability. While prior work establishes the expressivity benefits of LRNNs over transformers, it is unclear what makes LRNNs -- but not traditional, nonlinear RNNs -- as easy to parallelize in practice as transformers. We answer this question by providing a tight connection between types of RNNs and standard complexity classes. We show that LRNNs can be viewed as log-depth (bounded fan-in) arithmetic circuits, which represents only a slight depth overhead relative to log-depth boolean circuits that transformers admit. Furthermore, we show that nonlinear RNNs can solve $\mathsf{L}$-complete problems (and even $\mathsf{P}$-complete ones, under polynomial precision), revealing a fundamental barrier to parallelizing them as efficiently as transformers. Our theory also identifies fine-grained expressivity differences between recent popular LRNN variants: permutation-diagonal LRNNs are $\mathsf{NC}^1$-complete whereas diagonal-plus-low-rank LRNNs are more expressive ($\mathsf{PNC}^1$-complete). We provide further insight by associating each type of RNN with a corresponding automata-theoretic model that it can simulate. Together, our results reveal fundamental tradeoffs between nonlinear RNNs and different variants of LRNNs, providing a foundation for designing LLM architectures that achieve an optimal balance between expressivity and parallelism.
