Search papers, labs, and topics across Lattice.
This paper introduces the Hierarchical Autoregressive Transformer (HAT), a novel architecture that replaces tokenization with byte-level processing in LLMs. HAT uses an encoder transformer to aggregate bytes into word embeddings, which are then processed by a standard autoregressive transformer backbone, followed by a decoder that converts the backbone's output back into bytes. By converting Llama 3.1 8B and 70B models into the HAT architecture and training a 7B HAT model from scratch, the authors demonstrate improved text compression, robustness to spelling variations, and strong performance in English and German, often surpassing the original Llama 3.1.
Ditch the tokenizer: this new LLM architecture processes text at the byte level, offering better compression, spelling robustness, and multilingual performance.
Tokenization is a central component of natural language processing in current large language models (LLMs), enabling models to convert raw text into processable units. Although learned tokenizers are widely adopted, they exhibit notable limitations, including their large, fixed vocabulary sizes and poor adaptability to new domains or languages. We present a family of models with up to 70 billion parameters based on the hierarchical autoregressive transformer (HAT) architecture. In HAT, an encoder transformer aggregates bytes into word embeddings and then feeds them to the backbone, a classical autoregressive transformer. The outputs of the backbone are then cross-attended by the decoder and converted back into bytes. We show that we can reuse available pre-trained models by converting the Llama 3.1 8B and 70B models into the HAT architecture: Llama-3.1-8B-TFree-HAT and Llama-3.1-70B-TFree-HAT are byte-level models whose encoder and decoder are trained from scratch, but where we adapt the pre-trained Llama backbone, i.e., the transformer blocks with the embedding matrix and head removed, to handle word embeddings instead of the original tokens. We also provide a 7B HAT model, Llama-TFree-HAT-Pretrained, trained entirely from scratch on nearly 4 trillion words. The HAT architecture improves text compression by reducing the number of required sequence positions and enhances robustness to intra-word variations, e.g., spelling differences. Through pre-training, as well as subsequent supervised fine-tuning and direct preference optimization in English and German, we show strong proficiency in both languages, improving on the original Llama 3.1 in most benchmarks. We release our models (including 200 pre-training checkpoints) on Hugging Face.
