Search papers, labs, and topics across Lattice.
This paper introduces MultiHashFormer, a novel framework for hash-based autoregression in language models that addresses the limitations of many-to-one token collisions in causal LMs. By representing each token with a unique hash signature generated from multiple independent hash functions, the model achieves parameter efficiency while maintaining performance. Evaluated across various parameter scales, MultiHashFormer consistently outperforms standard Transformer models and effectively manages multilingual vocabulary expansion without increasing the parameter footprint.
MultiHashFormer achieves superior performance over traditional Transformers while maintaining a constant parameter footprint, even with multilingual vocabulary expansion.
Language models (LMs) represent tokens using embedding matrices that scale linearly with the vocabulary size. To constrain the parameter footprint, prior work proposes hashing many tokens into a single vector within encoder-only models. While this offers parameter efficiency, many-to-one collisions prevent its use in causal LMs. In this paper, we propose MultiHashFormer, a new framework that allows hash-based autoregression. Each token is represented as a unique hash signature, a short sequence of discrete hash IDs, generated by multiple independent hash functions. A Hash Encoder compresses this signature into a single latent vector for processing by a Transformer decoder. Then, a Hash Decoder generates the hash signature of the next token, which is then mapped back to text. We evaluate our approach at the 100M, 1B and 3B parameter scales, demonstrating that MultiHashFormer consistently outperforms standard Transformer LMs across multiple benchmarks. Furthermore, we show that our model handles multilingual vocabulary expansion with a constant parameter footprint without any modifications.
