Search papers, labs, and topics across Lattice.
The paper introduces a self-distillation method, GATES, designed for scenarios with unreliable supervision, specifically document-grounded question answering where the tutor has access to a document during training while the student does not. GATES leverages tutor consensus by sampling multiple reasoning traces and using agreement as a reliability signal to gate the distillation process, thereby mitigating the impact of potentially incorrect tutor outputs. The method distills knowledge through full tutor reasoning trajectories conditioned on the consensus signal, leading to significant improvements in both in-domain accuracy (46.0% to 62.0%) and performance on public math benchmarks (20.2% to 35.4%).
Ditch the assumption of tutor correctness: GATES leverages agreement between multiple reasoning traces to distill knowledge in document-grounded QA, boosting accuracy by over 15% on in-domain tasks.
We study self-distillation in settings where supervision is unreliable: there are no ground truth labels, verifiable rewards, or external graders to evaluate answers. We focus on document-grounded question answering with asymmetric context, where a single model serves as both tutor (with access to a relevant source document during training) and student (answering from the question alone at test time). Rather than assuming tutor correctness, we derive supervision online from tutor consensus by sampling multiple document-grounded reasoning traces and using agreement to gate learning. Conditioned on this reliability signal, we distill knowledge through full tutor reasoning trajectories (not just final answers), providing a dense and stable learning signal. Empirically, this consensus-gated trajectory distillation substantially improves transfer to the document-free student. Held-out in-domain accuracy under asymmetric evaluation improves from 46.0\% to 62.0\%, and average (maj@8) accuracy on public document-free math benchmarks improves from 20.2\% to 35.4\%.
