Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces an explicit word boundary marker system for subword tokenization, addressing the issue of duplicate embeddings caused by variations in word forms due to leading spaces and capitalization. By using boundary markers to delimit words and represent spaces, the proposed method reduces the number of unique token entries while maintaining similar tokenization compression levels. The results indicate that this approach enhances language modeling performance, achieving lower bits per byte across various marker schemes compared to the baseline.
Eliminating duplicate token entries with explicit boundary markers significantly boosts language modeling performance while maintaining compression efficiency.
Subword tokenizers represent many common words twice in space-using writing systems, once with a leading space and once without. The two entries have separate embeddings in models, so occurrences of one word are divided across rows that are trained independently, and the two forms need not even segment the string the same way: " together" may be a single entry while the same word without a preceding space is tokenized as "to|gether". Capitalization divides a word further, into as many as six forms. We introduce an alternative to standard whitespace conventions using an explicit word boundary marker, which prevents such duplication. Words are delimited by the boundary markers, and spaces between words are represented as pairs of such markers. Two shift codes do the same for title case and upper case, allowing one internal representation of a word to be re-used across different settings. Switching to this convention mitigates the duplicate-entry issue, but does not improve tokenization compression: for both vocabulary-learning algorithms, the best marker scheme stays within one percent of the baseline in characters per token, averaged across six languages. It does result in better language modeling performance. Every marker scheme tested downstream reaches lower bits per byte than the baseline, suggesting that duplication carries a cost that compression does not capture.