Search papers, labs, and topics across Lattice.
This paper introduces REER-PT, a novel framework that enhances pre-training data for language models by incorporating reverse-engineered reasoning annotations to clarify the connections between context and continuations. By leveraging perplexity as an optimization signal, REER-PT identifies and refines challenging continuations, resulting in a more informative training dataset while maintaining compatibility with conventional next-token prediction. The application of REER-PT leads to significant perplexity reductions and improves model performance by up to 2.07 percentage points on various knowledge and reasoning benchmarks.
Transforming pre-training data with reasoning annotations boosts model performance by over 2 percentage points without altering the training objective.
As language-model compute continues to scale, high-quality training data is becoming an increasingly important bottleneck. Conventional next-token prediction supervises what follows a context but leaves the intermediate reasoning behind that continuation implicit. We introduce \textbf{REER-PT}, a scalable framework that extends Reverse-Engineered Reasoning (REER) to raw pre-training data. REER-PT identifies continuations that are difficult to predict but can still be inferred from the preceding context, and inserts concise reasoning annotations that reconstruct the missing connection between context and continuation. Candidate annotations are generated and refined offline, with perplexity serving as the optimization signal. Constraints on length and target leakage filter out unhelpful or trivial annotations. This sparse transformation preserves the source text and remains compatible with standard next-token prediction, avoiding online reasoning rollouts during pre-training. We apply REER-PT to transform a source pre-training corpus into an augmented one. Across augmented-data, original-token, and selected-continuation comparisons, perplexity reductions range from 0.42 to 7.29, and only about 0.05\% of annotation 13-grams appear verbatim in the source text. We then train two 680M-parameter models with the same architecture and training configuration on the source and augmented corpora, respectively. The augmented-data model gains up to 2.07 percentage points on several knowledge and reasoning benchmarks. Together, the perplexity analysis indicates improved continuation predictability, while the controlled pre-training experiments suggest that this augmentation can improve model performance without changing the standard pre-training objective.
