Search papers, labs, and topics across Lattice.
This paper introduces Neuron On-Policy Self-Distillation (Neuron-OPSD), a novel framework for annotation-free self-distillation of large language models (LLMs) that utilizes internal neuron activations for data selection and teacher context construction. By avoiding the pitfalls of traditional self-evolution methods, Neuron-OPSD enhances in-domain performance while maintaining cross-domain generalization and reducing calibration errors. The approach demonstrates significant improvements on specialized-domain benchmarks, making it a viable solution for scenarios where expert annotations are scarce or expensive.
Leveraging internal neuron activations, Neuron-OPSD achieves superior in-domain performance without the need for costly expert annotations.
Post-training large language models (LLMs) without real-world interaction feedback or human-labeled supervision remains challenging, particularly in specialized domains where expert annotations are costly to obtain. Recent annotation-free self-evolution methods address this by using the model's own outputs as supervision signals, constructing a teacher via additional context and aggregating predictions across multiple rollouts through majority voting to produce pseudo-labels. However, these approaches are not without drawbacks: SFT- and GRPO-based variants suffer out-of-domain performance degradation, while reward-based on-policy RL inflates calibration error. In this paper, we propose Neuron On-Policy Self-Distillation (Neuron-OPSD), a data-centric framework for annotation-free self-distillation that leverages internal neuron activations to guide both training-data selection and teacher context construction. The model is then trained via on-policy distillation from the teacher distribution, requiring no ground-truth labels at any stage. Across specialized-domain benchmarks, Neuron-OPSD improves in-domain task performance while preserving cross-domain generalization and mitigating calibration collapse over prior annotation-free baselines. This framework is particularly relevant to settings where online interaction or external supervision is costly or infeasible, and is conceptually distinct from offline RL approaches that rely on logged, reward-labeled trajectories.
