Track genomics, neuroscience and molecular biology.
Search papers, labs, and topics across Lattice.
Off-policy updates in LLM RL can lead to policy drift that is masked by standard sequence-level masking techniques due to cancellation of token log-ratios. CARM addresses this by taking the absolute value of token log-ratios before averaging, preventing opposing probability changes from canceling and ensuring a more accurate reflection of policy drift. Experiments demonstrate CARM's superiority over geometric-mean masking, yielding significant improvements in mathematical reasoning and code generation benchmarks.
LLM RL policy updates can be significantly misrepresented by standard masking, leading to substantial unseen drift; CARM's absolute-value approach reveals and corrects this.
Recent years have witnessed the rapid adoption of reinforcement learning (RL) in large language model (LLM) post-training, with substantial gains in mathematical reasoning and code generation. In practical systems, however, policy updates and differences between rollout and training engines can make sampled responses off-policy. Sequence-level masking addresses this mismatch by deciding whether an entire response should contribute to optimization. A common masking rule uses the length-normalized geometric mean of sampled token probability ratios. Its signed log-ratios can cancel across positions, concealing substantial bidirectional policy drift. We propose \emph{Cancellation-Aware Response Masking} (CARM), a sequence-level mask that takes the absolute value of each token log-ratio before averaging, preventing opposing probability changes from canceling. We prove that accepted responses satisfy a joint bound on the fraction of sampled-token ratios outside a prescribed band and their mean log-distance beyond its boundaries. Experiments on mathematical reasoning and code generation show that CARM improves mean@16 averaged over AIME 2024/2025/2026 and BeyondAIME by up to $3.13$ percentage points over geometric-mean masking, and increases average pass@1 across four code benchmarks by $2.88$ points over the strongest evaluated baseline. These findings support CARM as a theoretically grounded and effective method for response-level off-policy control in LLM reinforcement learning.