Search papers, labs, and topics across Lattice.
This paper introduces Active-GRPO, a novel approach for molecular optimization that enhances the training of large language models by integrating adaptive imitation and self-improvement mechanisms. By allowing the policy to dynamically choose when to imitate reference data and when to leverage its own discoveries, Active-GRPO overcomes the limitations of traditional reinforcement learning methods that rely on sparse feedback. The results demonstrate significant improvements in performance metrics, achieving an average SRxSim of 0.1773, surpassing previous benchmarks set by GRPO and RePO.
Active-GRPO not only outperforms existing methods in molecular optimization but also redefines how models can adaptively balance imitation and self-discovery during training.
Scientific reasoning is an increasingly important capability of large language models, yet improving the robustness and efficiency of training such reasoning remains a key open challenge. We study this problem in instruction-based molecular optimization, where answer-only supervised fine-tuning (SFT) collapses multi-step reasoning and reinforcement learning with verifiable rewards (RLVR) suffers from sparse feedback. Reference-guided Policy Optimization mitigates both by anchoring policy updates to dataset-provided references, but its effectiveness is tightly coupled to reference quality: weak or misaligned references impose a performance ceiling. To overcome this ceiling, we propose active reasoning, a paradigm in which the policy actively decides, on a per-instance basis, when to imitate a reference and when to reinforce its own discoveries, while continuously upgrading what it imitates. We instantiate this paradigm as Active Group Relative Policy Optimization (Active-GRPO), realized through two coupled mechanisms: active imitate-reinforce and active referencing. The former performs imitation learning when the reference still outperforms the policy's own candidates, and shifts to self-improvement via reinforcement learning once the policy has generated molecules that surpass the reference. The latter continuously upgrades the reference itself by replacing it with the best policy-generated candidate discovered so far, progressively raising the imitation target and ensuring that reference guidance remains informative-rather than restrictive-throughout training. Across TOMG-Bench MOLOPT, Active-GRPO improves average SRxSim from 0.0959 for GRPO and 0.1665 for RePO to 0.1773 under matched three-seed evaluation, with statistically significant gains on LogP, MR, and QED.
Stanford HAI