Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Universal Byte-Level Encoding (UBE), a novel byte-level representation that re-encodes multilingual codepoints into a fixed two-byte UTF-16 format to improve boundary prediction for hierarchical byte-level language models. UBE significantly reduces grapheme-boundary prediction errors across numerous languages and scripts by ensuring the first byte carries more distinguishing codepoint information, unlike UTF-8 where initial bytes can be ambiguous. The key result is a dramatic reduction in boundary prediction error, particularly for languages with complex grapheme structures, enabling more accurate and efficient processing of multilingual text at the byte level.
A simple byte reordering strategy slashes multilingual boundary prediction errors by over 70% in scripts like Korean and Chinese, fundamentally altering how byte-level models perceive codepoint transitions.
Many hierarchical byte-level language models group bytes into patches using a learned boundary predictor whose view of the next codepoint depends on encoding and lookahead. In UTF-8, codepoints of many scripts take three bytes: 8 of 24 bits mark length and continuation, and the first byte carries only 4 high codepoint bits. After the same left context, a mark extending a grapheme cluster and a letter starting the next can thus share their first byte. Universal Byte-Level Encoding (UBE) keeps one- and two-byte UTF-8 encodings but writes these codepoints as two UTF-16 bytes, low byte first, so its first symbol carries 8 low codepoint bits that can distinguish such continuations. We train small causal predictors on matched FineWeb-2 text and evaluate on FLORES-200 under UTF-8, UBE, and UBE's same-length big-endian variant. At one-symbol lookahead, balanced grapheme-boundary error at codepoint boundaries across 17 languages in 13 Brahmic scripts is 8.76% with UTF-8, 8.62% with the big-endian variant, and 0.06% with UBE. When the target instead retains a parent tokenizer's subword cuts at grapheme boundaries, UBE lowers the mean error relative to UTF-8 at the same lookahead by 72.91% in Korean, 35.29% in Chinese, and 67.32-99.55% in eight more scripts. In the seven of them whose main Unicode block fits on one 256-codepoint page, the big-endian variant stays within 3.0% of UTF-8, so the gain follows byte order, not length. In Korean and Chinese, UBE's mean error falls below UTF-8's final mean error after only 25% and 37.5% of training. Which byte comes first thus affects how much lookahead a byte-level boundary predictor needs.