Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces MLP-LDRU, a novel Log-Depth Recurrent Unit architecture designed to mitigate positional biases and fixed computational depth limitations in recurrent and transformer models, respectively. MLP-LDRU leverages associativity-biased operators to approximate recurrence through parallel reduction, enabling efficient processing of variable-length sequences. Empirical results on 21 regular language tasks demonstrate near-perfect out-of-distribution accuracy, surpassing existing recurrent and attention-based models, with competitive performance also shown on ListOps and NLP classification tasks.
Achieve perfect length generalization on complex regular languages with a novel recurrent unit that approximates recurrence through parallel reduction, outperforming transformers and standard RNNs.
Length generalization remains a persistent challenge for neural networks: recurrent models tend to suffer from positional biases, while transformers are constrained by fixed computational depth. Regular languages provide a frequently used testbed for evaluating length generalization, as label prediction can be checked for any sequence length. We propose MLP-LDRU, a type of Log-Depth Recurrent Unit, which captures a class of associativity-biased operators designed to approximate recurrence through parallel reduction. We evaluate MLP-LDRU on 21 regular-language tasks, consisting of standard benchmarks and new prefix languages, where it achieves 100% out-of-distribution accuracy on 18 tasks and at least 99.9% on the remaining 3 when increasing max training length, outperforming comparable recurrent and attention-based models. We further evaluate MLP-LDRU beyond regular languages on ListOps and NLP classification benchmarks, where it performs competitively.