Search papers, labs, and topics across Lattice.
This paper explores the structural advantages of byte-level language models, particularly their ability to share an output space, facilitating exact knowledge transfer regardless of tokenization methods. The authors hypothesize that the distributions produced by these models can be disentangled, allowing for independent adjustments of capabilities and boundary placements. Preliminary experiments are proposed to validate this hypothesis, suggesting that adopting a byte-level interface could revolutionize model interoperability and capability transfer in the AI community.
Byte-level models could enable seamless capability transfer and boundary reshaping, making model enhancements as easy as swapping out components.
Byte-level language models are usually argued for on the grounds of robustness, multilingual fairness, and character-level skills. We point to a different, structural advantage: because they read and write bytes, any two of them share an output space, so knowledge transfer between them is exact and independent of how either was originally tokenized. We hypothesize that the two distributions a byte-level model produces, one over the next byte, one over where its patch boundaries fall, can be disentangled and changed almost independently. A model could absorb a teacher's capability while keeping its own boundaries, or change how it places those boundaries while keeping its capabilities. We lay out the two experiments that would settle the hypothesis, alongside preliminary measurements of the properties they rest on. We argue that the community should move toward a byte-level interface as a shared standard: if the hypothesis holds, then once byte-level models are the norm, transferring capabilities and reshaping boundaries between them become cheap and routine, free of the per-model tokenizer that blocks them today.
