Search papers, labs, and topics across Lattice.
This paper investigates the trainability of NLI checkers used as rewards in RL-based medical RAG agents. It finds that the checker's output distribution during training, rather than its held-out accuracy, determines gradient quality, identifying signal collapse and reward hacking as key failure modes. The study demonstrates that a moderate-signal local classifier can outperform a strong proprietary checker due to the latter's tendency to induce reward hacking.
Seemingly strong NLI checkers can actually *hurt* medical RAG training by collapsing the RL gradient or triggering reward-hacking cascades like ultra-short answers and search avoidance.
Medical RAG needs evidence-grounded claims, so plugging a claim-level NLI checker into retrieval-augmented RL is intuitive. \textbf{We find that the checker's \emph{output distribution} during training, not its held-out accuracy, decides whether it provides trainable gradient.} We compare four NLI checker back-ends as process rewards inside a GRPO-trained medical RAG agent (Qwen2.5-7B, replicated on Qwen3-4B and Llama-3.1-8B) across four held-out medical QA benchmarks. Three diagnostic findings emerge. \textbf{(i)} Signal collapse is log-prob-specific: LLM log-probability scoring labels over 97\% of claims neutral -- collapsing the RL gradient to zero -- while a calibrated MedNLI classifier scores the same pairs non-degenerately. \textbf{(ii)} Moderate signal beats strong signal on answer quality: a strong proprietary checker triggers a three-step reward-hacking cascade -- ultra-short answers, search avoidance, language collapse -- so a moderate-signal local classifier trains a higher-quality model (\textbf{+12\% BERTScore over zero-shot, no GPT dependency}). \textbf{(iii)} Signal strength is policy-dependent: the same checker registers as moderate on one policy but strong on another without triggering the cascade end-state. We frame these as boundary conditions for verifier-as-reward systems.
