Search papers, labs, and topics across Lattice.
This study introduces Quantile-Guided Density Estimation (QGDE) to estimate the corpus ratios of arbitrary target tokens from released tokenizer vocabularies, addressing the challenge of hidden pretraining corpus composition. By leveraging stable token ID鈥搑atio distributions across different corpora, the method achieves impressive accuracy, with mean relative errors as low as 3.00% for token-level estimates. The findings highlight the potential of released vocabularies to provide fine-grained insights into corpus composition, enhancing our understanding of LLM capabilities.
Released tokenizer vocabularies can yield precise estimates of hidden corpus compositions, revealing insights that were previously obscured.
Pretraining corpus composition shapes LLM capabilities, but it often remains hidden even when model weights are released. Prior work has inferred corpus mixtures or traced specific token groups from released tokenizer vocabularies; in contrast, we estimate corpus ratios for arbitrary target tokens. We first show that BPE tokenizers trained on different corpora share stable token ID--ratio distributions, motivating distribution transfer from known corpora to a target tokenizer trained on hidden corpora. We then propose Quantile-Guided Density Estimation (QGDE), which approximates this distribution with multiple quantile trends and uses local density weighting to produce token-level estimates. In controlled settings and a realistic setting using the released SmolLM tokenizer, QGDE achieves mean relative errors as low as 3.00% for token-level estimation and 3.08% after aggregation into category-level mixtures. These results suggest that released tokenizer vocabularies provide a useful signal for fine-grained corpus estimation beyond coarse composition inference.
DAMO
Tsinghua AI