Search papers, labs, and topics across Lattice.
This paper introduces KDoS (Knowledge Distribution-optimized Synthesis), a novel framework for knowledge injection in Large Language Models (LLMs) that optimizes the distribution of synthetic data to enhance knowledge boundaries. By employing a three-stage feedback mechanism, KDoS shifts the focus from arbitrary token counts to strategically optimized knowledge density, resulting in significant improvements in knowledge expansion. Experiments across various model sizes and data scales demonstrate that KDoS consistently outperforms existing methods, revealing a stable optimal knowledge distribution that maximizes boundary expansion across different architectures.
An optimal knowledge distribution can significantly enhance LLM knowledge boundaries, outperforming traditional synthesis methods across multiple benchmarks.
Knowledge injection via synthetic data is crucial for enhancing Large Language Models (LLMs). However, current synthesis methods simply stop at preset token counts or fixed data ratios, lacking awareness of knowledge distribution. This results in some domains being sparse while others are redundant, limiting LLM knowledge boundaries. We revisit knowledge injection from a distribution perspective and hypothesize that an optimal knowledge distribution exists to maximize knowledge boundary expansion. We propose KDoS (Knowledge Distribution-optimized Synthesis), a framework that introduces knowledge density to drive synthesis through a three-stage feedback mechanism, shifting from blind generation to distribution-optimized synthesis. We construct Wikipedia-based synthetic data with varying knowledge distributions and conduct experiments on models from 0.6B to 16B (Qwen, Ling, LLaMA) and data scales from 1B to 5B tokens. Our key findings are: (1) an optimal knowledge distribution consistently maximizes boundary expansion; (2) this distribution is stable across backbones and scales; (3) KDoS outperforms baselines across six knowledge benchmarks. Our work offers a new perspective and practical framework for synthetic data-driven knowledge injection.
