Search papers, labs, and topics across Lattice.
This paper introduces Gold-Action Policy Distillation (GAPD), a novel training framework for agentic reinforcement learning in knowledge base question answering (KBQA) that leverages gold logical forms for dense token-level guidance. GAPD uses MID-ANCHOR MATCHING to align student and gold action sequences by treating intermediate entities as state anchors and distilling the teacher's token distribution back to the student policy. Experiments demonstrate that GAPD consistently achieves state-of-the-art results on WebQSP, GrailQA, and GraphQ, showcasing the effectiveness of dense action-level supervision in RL-based KBQA.
Achieve state-of-the-art results in agentic knowledge base question answering by distilling gold-action policies into on-policy student rollouts, bridging the gap between sparse rewards and weakly supervised intermediate actions.
Reinforcement learning (RL) is a natural fit for agentic knowledge base question answering (KBQA), where a model must issue executable actions, observe knowledge-base feedback, and eventually return an answer. However, current RL-based KBQA systems mainly optimize sparse rewards from the final answer, leaving intermediate action errors weakly supervised. This is especially limiting for logical-form annotated KBQA benchmarks: gold logical forms can be converted into executable action sequences, but existing pipelines use them mainly for warm-start data construction rather than for on-policy RL updates. We propose GAPD, a training-time Gold-Action Policy Distillation framework that adds dense token-level guidance to outcome-based RL. To align gold actions with on-policy student rollouts, GAPD uses MID-ANCHOR MATCHING: it treats the intermediate entities reached during student exploration and gold execution as state anchors, and matches student states to gold states through these explored entity sets. The current policy conditioned on this aligned gold action serves as a stop-gradient teacher, whose token distribution is distilled back to the ordinary student policy over generated action-token spans. GAPD consistently surpasses the current state of the art on WebQSP, GrailQA, and GraphQ.
