Search papers, labs, and topics across Lattice.
This paper addresses the challenge of aligning token-level generation in autoregressive document retrieval with document-level relevance by introducing a fine-grained reinforcement learning framework that utilizes token-level relevance rewards. By estimating step-wise rewards based on the impact of individual token decisions on retrieval quality, the proposed method allows for more precise credit assignment compared to traditional sequence-level reward approaches. Experimental results on retrieval benchmarks reveal that this fine-grained supervision significantly enhances retrieval effectiveness, outperforming existing baselines.
Token-level credit assignment can drastically improve the effectiveness of generative document retrieval, leading to better alignment between generation and relevance.
Generative retrieval models perform document retrieval by autoregressively generating document identifiers (DocIDs). This process naturally forms a sequential decision problem, where each decoding step selects a DocID token and the complete token sequence determines the retrieved document. However, retrieval effectiveness is typically evaluated only after the full DocID is generated, creating a mismatch between token-level generation and document-level relevance supervision. As a result, existing reinforcement learning methods for generative retrieval mostly rely on sequence-level rewards, where the same document-level feedback is propagated to all decoding steps. Such coarse-grained feedback makes it difficult to identify which token decisions are responsible for successful or failed retrieval. In this work, we propose a fine-grained reinforcement learning framework for generative retrieval with token-level relevance rewards. Instead of assigning a single reward to the entire generated DocID, we estimate step-wise rewards by measuring how each token decision changes the expected retrieval quality of the corresponding generation trajectory. This enables more precise credit assignment and encourages the policy to favor token decisions that contribute more directly to document-level relevance. We further develop practical reward estimation strategies tailored to the DocID generation process and incorporate them into a policy optimization framework. Experiments on retrieval benchmarks show that our method consistently outperforms sequence-level reward baselines, demonstrating the effectiveness of fine-grained supervision for aligning autoregressive DocID generation with retrieval objectives.
Tsinghua AI