Search papers, labs, and topics across Lattice.
This paper introduces LEGO-RL, a novel framework designed to enhance the performance of coding agents by aligning their native execution environments with scalable policy-gradient optimization. By addressing issues such as environmental crashes and reward hacking, LEGO-RL employs in-process LLM proxying, scalable sandbox orchestration, and an integrated monitoring plugin to ensure reliable and observable training. The evaluation shows significant performance improvements in the sparse MoE model Qwen3.5-35B-A3B across multiple coding-agent harnesses, achieving up to a 8.4% increase in task success rates while maintaining high rollout-training correlation.
LEGO-RL boosts coding agent performance by up to 8.4% while ensuring robust training signals and execution reliability.
Reinforcement learning for coding agents increasingly relies on long-running agent harnesses to manage tool integration, repository contexts, and execution feedback. However, the native execution environments of these harnesses are inherently misaligned with policy-gradient training: environmental crashes and reward hacking corrupt outcome signals, while train-inference discrepancies decouple rollout behavior from policy updates. To address this, we present LEGO-RL, a framework that bridges native coding-agent harnesses with scalable policy-gradient optimization without modifying their internal control flow. LEGO-RL is built upon three pillars: (1) faithful optimization via in-process LLM proxying that captures raw generation streams for token-level alignment and robust trainer-side log-probability recomputation, even under harness-side compaction or re-serialization; (2) reliable execution via scalable sandbox orchestration featuring image caching and stage-wise defenses to mitigate reward hacking; and (3) observable training through an integrated plugin that automates validation and monitoring, paired with a Live UI for granular trajectory diagnostics. We evaluate LEGO-RL by training the sparse MoE model Qwen3.5-35B-A3B with GSPO across three native coding-agent harnesses. LEGO-RL improves Qwen3.5-35B-A3B across OpenHands SDK (64.0% to 70.4%), Claude Code (62.4% to 68.2%), and OpenCode (57.2% to 66.6%) on SWE-bench Verified, while maintaining a rollout-training probability correlation above 0.99.
