Search papers, labs, and topics across Lattice.
This paper introduces SAERL, a data engineering framework for LLM reinforcement learning that leverages Sparse Autoencoders (SAEs) to extract intrinsic data properties (diversity, difficulty, quality) from model internals. These properties are then used to guide data engineering operations like batch mixing, curriculum learning, and data filtering. Experiments on Qwen2.5-Math-1.5B show SAERL improves average accuracy by 3% over vanilla GRPO and reaches target accuracy with 20% fewer training steps, demonstrating the effectiveness and transferability of SAEs for data engineering.
Forget external signals – unlock better LLM post-training by mining model internals with sparse autoencoders to reveal data diversity, difficulty, and quality.
Model internals encode rich information about how a large language model (LLM) processes its training data; however, post-training data engineering largely relies on external signals and ignores rich intrinsic signals lying in model internals. We propose SAERL, a data engineering framework for LLM reinforcement learning (RL). It models three intrinsic data properties: diversity, difficulty, and quality, using model internals extracted with Sparse Autoencoder (SAE), an advanced mechanistic interpretability tool. Each property grounds a concrete data engineering operation: SAE-space clustering with moderate batch mixing for batch diversity control, a difficulty proxy for easy-to-hard curriculum ordering, and a quality probe for data filtering. SAERL improves average accuracy by 3.00% over vanilla GRPO and reaches target accuracy with 20% fewer training steps on Qwen2.5-Math-1.5B, with consistent gains across model scales and RL algorithms. Experiments show that SAE transfers effectively across model families and scales, serving as a lightweight and reusable data engineering tool. These results demonstrate that model internals are a powerful and practical source of signals for post-training data engineering.
Tsinghua AI