Search papers, labs, and topics across Lattice.
This paper reformulates the autoregressive, attention-free Avey architecture into an encoder-only model suitable for resource-constrained NLP applications. They introduce decoupled static/dynamic parameterizations, stability-oriented normalization, and neural compression techniques to improve Avey's performance. The resulting Avey-B model outperforms Transformer-based encoders on token classification and information retrieval tasks, demonstrating improved efficiency and scalability to longer contexts.
Attention-free Avey-B matches or exceeds the performance of BERT-style encoders on token classification and information retrieval, offering a more efficient alternative for resource-constrained NLP applications.
Compact pretrained bidirectional encoders remain the backbone of industrial NLP under tight compute and memory budgets. Their effectiveness stems from self-attention's ability to deliver high-quality bidirectional contextualization with sequence-level parallelism, as popularized by BERT-style architectures. Recently, Avey was introduced as an autoregressive, attention-free alternative that naturally admits an encoder-only adaptation. In this paper, we reformulate Avey for the encoder-only paradigm and propose several innovations to its architecture, including decoupled static and dynamic parameterizations, stability-oriented normalization, and neural compression. Results show that this reformulated architecture compares favorably to four widely used Transformer-based encoders, consistently outperforming them on standard token-classification and information-retrieval benchmarks while scaling more efficiently to long contexts.
