Search papers, labs, and topics across Lattice.
This study investigates whether reinforcement learning (RL) post-training can create new compositional reasoning strategies from primitive skills in a Transformer model, rather than merely amplifying existing capabilities. Conducted in a controlled rewrite-grammar environment, the research reveals that RL effectively solves complex problems that the pretrained model struggles with, utilizing a phased compositional mechanism that reorganizes and consolidates primitive competencies into reusable strategies. Key findings indicate that RL's selective exploration leads to valid and stable procedural compositions, contrasting sharply with the shortcut-like rewrites produced by rejection fine-tuning.
RL post-training not only amplifies existing skills but also synthesizes them into robust, reusable reasoning strategies that outperform traditional methods.
Does RL post-training merely amplify primitive skills already latent in a base model, or can it compose primitive skills into new higher-level strategies? We study this question in a fully observable rewrite-grammar environment where the pretraining distribution is known and every generated rewrite can be audited. A Transformer is pretrained on primitive symbol-rewrite chains and post-trained on a Trace-based reasoning task with only a binary final-answer reward. RL solves held-out problems that remain rarely solved by the pretrained model even under much larger sampling budgets, while rejection fine-tuning improves early but plateaus. Trace analysis shows that RL reorganizes primitive competence through a phased compositional mechanism: it first strengthens primitive reductions, then discovers valid composed procedures. These include sequential compositions, which collapse ordered chains of primitive contractions, and parallel compositions, which combine independent primitive contractions in a single step. The composed procedures are not isolated samples; they are reused and consolidated into a stable repertoire. Comparing RL with rejection fine-tuning shows that the key difference is not exploration volume but selectivity: RFT produces many shortcut-like rewrites, much of them invalid, whereas RL concentrates exploration into valid reusable structure. Pretraining ablations show that the emergence of compositional strategies is gated not by primitive exposure alone, but by whether pretraining organizes primitive competence into reduction procedures that RL can later compress. The base model provides weak procedural ingredients; RL builds them into reliable higher-level strategies.
