Search papers, labs, and topics across Lattice.
The paper introduces PCMind-2.1-Kaiyuan-2B, a 2B parameter open-source LLM, designed to improve training efficiency under resource constraints. They employ a Quantile Data Benchmarking method for data mixing, Strategic Selective Repetition for high-quality data leverage, and a Multi-Domain Curriculum Training policy for sample ordering. Kaiyuan-2B achieves competitive performance with state-of-the-art open-source models while using optimized data preprocessing and architectural modifications for FP16 stability.
Resource-constrained LLM pretraining gets a boost with PCMind-2.1-Kaiyuan-2B's open-source release and novel techniques for data mixing, repetition, and curriculum learning.
The rapid advancement of Large Language Models (LLMs) has resulted in a significant knowledge gap between the open-source community and industry, primarily because the latter relies on closed-source, high-quality data and training recipes. To address this, we introduce PCMind-2.1-Kaiyuan-2B, a fully open-source 2-billion-parameter model focused on improving training efficiency and effectiveness under resource constraints. Our methodology includes three key innovations: a Quantile Data Benchmarking method for systematically comparing heterogeneous open-source datasets and providing insights on data mixing strategies; a Strategic Selective Repetition scheme within a multi-phase paradigm to effectively leverage sparse, high-quality data; and a Multi-Domain Curriculum Training policy that orders samples by quality. Supported by a highly optimized data preprocessing pipeline and architectural modifications for FP16 stability, Kaiyuan-2B achieves performance competitive with state-of-the-art fully open-source models, demonstrating practical and scalable solutions for resource-limited pretraining. We release all assets (including model weights, data, and code) under Apache 2.0 license at https://huggingface.co/thu-pacman/PCMind-2.1-Kaiyuan-2B.
