Search papers, labs, and topics across Lattice.
This paper introduces a circuit-grounded framework that integrates mechanistic interpretability with data generation, moving beyond heuristic prompt-based controls. By identifying specialized model-internal circuits that influence data quality along learnability, challenge, and alignment axes, the authors develop a method for actively steering data generation. Their approach, SAMS (Stage-Aware Mechanistic Scheduling), enhances the diversity and utility of generated data, leading to improved performance on multiple-choice QA tasks compared to traditional methods.
Mechanistic control over data generation reveals hidden model dynamics, leading to more diverse and effective datasets that enhance downstream performance.
While recent advances in data synthesis aim to curate high-quality datasets, most generation pipelines still rely on heuristic prompt-based control. This black-box paradigm provides limited insight into how individual samples interact with a model's underlying learning dynamics. To bridge this gap, we propose a circuit-grounded framework that connects training-dynamics-based data valuation with mechanistic interpretability (MI). Specifically, we conceptualize data quality along three complementary utility axes, learnability, challenge, and alignment. First, we uncover specialized model-internal circuits that causally govern these utility signals. Then, moving beyond heuristic prompting toward mechanistic control, we leverage these circuits as controllable interfaces, actively steering generation to produce utility-targeted data. Building on this capability, we introduce SAMS (Stage-Aware Mechanistic Scheduling), which schedules circuit-steered data according to the model's evolving optimization needs. Experiments on multiple-choice QA tasks demonstrate that our approach yields precisely controlled data with greater diversity than prompt-based baselines, consistently improving downstream performance and calibration. Ultimately, this work establishes a principled white-box paradigm for interpretable data generation, pioneering the use of MI not just as an analytical tool, but as a practical, controllable interface.
