Search papers, labs, and topics across Lattice.
This paper introduces MetaRAG, a belief-action aligned policy optimization framework designed to enhance agentic retrieval-augmented generation (RAG) by integrating an internal belief mechanism into the decision-making process. By reformulating the search decision quality through belief-action alignment, MetaRAG employs Verify-first Action Generation and Internal Belief Probing to improve the accuracy and efficiency of responses while avoiding the reinforcement of incorrect trajectories. Experimental results across seven public QA benchmarks demonstrate that MetaRAG significantly outperforms existing RL-based methods, showcasing its adaptability across various research settings and model architectures.
MetaRAG achieves a superior accuracy-efficiency trade-off in agentic RAG by aligning decision-making with the model's internal beliefs, outperforming traditional RL methods.
Agentic retrieval-augmented generation (RAG) requires language models to decide when to continue searching and when to answer. Existing RL-based methods rely on external supervision and overlook the agent's internal belief about whether the current evidence is sufficient. To address this problem, we reformulate the search decision quality as belief-action alignment and propose MetaRAG, a belief-action aligned policy optimization framework for agentic RAG. MetaRAG uses Verify-first Action Generation to elicit an explicit verification process before each actual action, and Internal Belief Probing to estimate the policy model's own answerability belief from the same question-history context. Based on these, MetaRAG derives a consistency reward that is further gated by answer correctness, avoiding reinforcement of internally consistent but incorrect trajectories. The belief probe is used only during training and introduces no inference-time overhead. Experiments on seven public QA benchmarks show that MetaRAG consistently improves the accuracy-efficiency trade-off over strong RL-based agentic RAG baselines, with gains that transfer to deep research settings, different optimizers, and multiple model backbones.
