Search papers, labs, and topics across Lattice.
This paper introduces Q-Steer, a novel action-value steering mechanism for molecular policy optimization that addresses the challenges posed by delayed feedback in oracle-limited settings. By employing an offline-trained prefix-action value scorer (PAVS-Q) to estimate the reward of potential next tokens, Q-Steer enhances the decision-making process during molecule generation without altering the existing optimizer update rules or online oracle budget. Experimental results demonstrate that Q-Steer consistently improves the mean valid-unique score across multiple molecular language models and optimizers, achieving significant performance gains in all tested configurations.
Q-Steer boosts molecular optimization rewards by leveraging action-value estimates during rollout, leading to consistent performance improvements across various models and optimizers.
Oracle-limited molecular optimization gives reward only after a complete molecule is generated, while each rollout requires many local next-token decisions. This delayed-feedback interface makes molecular policy optimization myopic: an optimizer can learn that a molecule was good without knowing which intermediate actions made it good. We introduce Q-Steer, a rollout-time action-value steering primitive for molecular language models. Q-Steer uses an offline-trained and frozen prefix-action value scorer, PAVS-Q, that estimates the downstream reward of taking a candidate next token under a partial SMILES prefix, then adds a normalized value bonus to sampling logits. The optimizer update rule and online oracle budget are unchanged; the claim is fixed-online-oracle performance, not equal total compute. On PMO23 with a fixed 10,000-call online budget, complete factorial studies across two molecular language-model backbones and four optimizers show that Q-Steer improves mean valid-unique score in all eight backbone-optimizer cells, with positive macro mean-score gains between +0.033 and +0.049 and 18-20 task wins per cell. Mechanism controls show that action identity matters: prefix-broadcast values are nearly neutral, while shuffled action values harm performance. These results support Q-Steer as a reusable rollout-time action-value wrapper that improves average molecular optimization reward across optimizer families and policy backbones without changing the online oracle budget.
