Search papers, labs, and topics across Lattice.
The authors reveal that standard sampling-based reinforcement learning via GRPO structurally fails in bounded tool-selection domains, where policy concentration causes reward collisions and leaves over 20% of training queries with zero advantage signal. To resolve this, they introduce Full-Group Policy Optimization (FGPO), which replaces sampled rollouts with exact expectation optimization over all tool subsets and decouples inference by precomputing rewards into an offline table. Across five frozen reasoners and three genomic benchmarks, FGPO outperforms GRPO in all 15 settings by an average of 6.75 points (and up to 14.20) while reducing frozen-reasoner evaluations by 2.4x.
Sampling rollouts via GRPO actively degrades tool-selection training as policies concentrate, whereas exact policy optimization over enumerable combinatorial tool spaces eliminates vanishing gradients and boosts accuracy by up to 14.2 points.
Reinforcement learning over a frozen reasoner has become a common recipe for teaching a policy which external tools to invoke. We show that this recipe becomes structurally mismatched in specialist scientific settings where the complete tool-subset space is enumerable. There, a small set of recurring computational capabilities covers the domain, so the space of tool subsets is combinatorial yet small enough to enumerate, and GRPO still estimates an action expectation from a handful of sampled rollouts. Worse, the approximation degrades as training succeeds: as the policy concentrates on preferred subsets it resamples them, sampled rewards collide, and the group-normalized advantage vanishes. On genomic reasoning the fraction of questions yielding no reward signal rises from 0.2% under a uniform reference policy to 20.8% after GRPO training. As a remedy, we introduce FGPO (Full-Group Policy Optimization), which (1) scores every tool subset and optimizes the exact action expectation, so each update sees the complete action space, and (2) precomputes the reward of each question--subset pair into an exhaustive table, removing frozen-reasoner calls from the training loop entirely. Across five frozen reasoners and three genomic benchmarks, FGPO outperforms GRPO in all 15 settings by 6.75 points on average and up to 14.20, while a standard on-demand GRPO schedule would require 2.4 times as many frozen-reasoner reward evaluations and, on GenomeQA, FGPO cuts invoked tools per question from 2.36 to 1.40.
