Search papers, labs, and topics across Lattice.
This paper introduces N-GRPO, an innovative exploration strategy that enhances policy optimization by employing Semantic Neighbor Mixing within the Group Relative Policy Optimization framework. By dynamically mixing embeddings of anchor tokens with their nearest semantic neighbors, N-GRPO effectively increases diversity in solution paths while maintaining semantic consistency, addressing the limitations of both token-level sampling and embedding-level noise. Experimental results demonstrate that N-GRPO significantly outperforms strong baselines on mathematical reasoning benchmarks and shows improved generalization on out-of-distribution tasks.
N-GRPO achieves superior performance in mathematical reasoning by mixing embeddings, leading to diverse yet semantically consistent solution paths.
The success of Large Language Models in mathematical reasoning relies heavily on the generation of diverse and valid solution paths during the rollout phase. However, current rollout techniques face a fundamental trade-off: token-level sampling often yields redundant trajectories that differ only in rephrasing, while embedding-level methods utilizing random noise frequently disrupt semantic consistency. To resolve this, we introduce N-GRPO, a novel exploration strategy integrated into the Group Relative Policy Optimization (GRPO) framework. Rather than relying on token-level sampling or native embedding-level noise, our approach leverages Semantic Neighbor Mixing. This mechanism dynamically constructs input representations by mixing the embeddings of an anchor token and its nearest semantic neighbors, thereby injecting diversity while strictly adhering to the local semantic manifold. Experimental evaluations on the DeepSeek-R1-Distill-Qwen models across different sizes show that N-GRPO not only achieves consistent improvements over strong baselines on math reasoning benchmarks but also exhibits robust generalization capabilities on out-of-distribution tasks.
