Search papers, labs, and topics across Lattice.
This paper introduces Adaptive Targeted Dynamic Chunking (ATDC), a curriculum learning-based method for optimizing byte compression ratios in tokenization-free hierarchical models. ATDC progressively increases the compression ratio during training, stabilizing learning and improving performance. Experiments on the FineWeb-Edu 100B dataset show that ATDC achieves competitive Bits-Per-Byte (BPB) performance and superior downstream task performance compared to fixed compression baselines.
Forget fixed chunk sizes: dynamically adjusting byte compression during training unlocks more stable and performant tokenization-free hierarchical models.
Tokenization-free hierarchical models are emerging as a promising alternative to traditional Large Language Models (LLMs), addressing inherent preprocessing issues such as vocabulary design complexity, out-of-vocabulary (OOV) errors, and language-specific constraints. However, a significant challenge in these byte-level methods is the optimization of the compression ratio, a critical factor that dictates model performance for processing bytes data via chunks. In this paper, we propose Adaptive Targeted Dynamic Chunking (ATDC), a novel byte-compression control mechanism designed to enhance the effectiveness of dynamic chunking within hierarchical architectures. Our approach utilizes curriculum learning to progressively adjust the compression ratio during training, transitioning from low to high compression to stabilize the learning process. We provide an analysis establishing the relationship between the target compression ratio and Bytes-Per-Innermost-Chunk (BPIC), allowing for tracking of chunk-size evolution throughout the training phase. Evaluations conducted on the FineWeb-Edu 100B dataset demonstrate that hierarchical models equipped with ATDC achieve competitive Bits-Per-Byte (BPB) performance compared to conventional baselines operating at both byte and token levels. Furthermore, the proposed method exhibits more stable training dynamics and superior final performance across diverse downstream tasks compared to models using fixed compression ratios, while maintaining the inherent robustness and flexibility of byte-level processing.
