Search papers, labs, and topics across Lattice.
ByteFlow Net, a novel hierarchical architecture, eliminates the need for tokenizers in language models by learning to segment raw byte streams into semantically meaningful units. This segmentation is driven by compression principles, specifically minimizing the coding rate of latent representations, and uses Top-$K$ selection to maintain a static computation graph. Experiments show that ByteFlow Net outperforms both BPE-based Transformers and existing byte-level architectures, demonstrating the effectiveness of tokenizer-free language modeling.
Ditch the tokenizer: ByteFlow Net learns to segment raw bytes into meaningful units via compression, outperforming traditional tokenization methods.
Modern language models still rely on fixed, pre-defined subword tokenizations. Once a tokenizer is trained, the LM can only operate at this fixed level of granularity, which often leads to brittle and counterintuitive behaviors even in otherwise strong reasoning models. We introduce \textbf{ByteFlow Net}, a new hierarchical architecture that removes tokenizers entirely and instead enables models to learn their own segmentation of raw byte streams into semantically meaningful units. ByteFlow Net performs compression-driven segmentation based on the coding rate of latent representations, yielding adaptive boundaries \emph{while preserving a static computation graph via Top-$K$ selection}. Unlike prior self-tokenizing methods that depend on brittle heuristics with human-designed inductive biases, ByteFlow Net adapts its internal representation granularity to the input itself. Experiments demonstrate that this compression-based chunking strategy yields substantial performance gains, with ByteFlow Net outperforming both BPE-based Transformers and previous byte-level architectures. These results suggest that end-to-end, tokenizer-free modeling is not only feasible but also more effective, opening a path toward more adaptive and information-grounded language models.
