Search papers, labs, and topics across Lattice.
This study investigates the impact of execution infrastructure on coding-agent reinforcement learning by comparing four different execution substrates: single containers, hosted sandboxes, Kubernetes-orchestrated containers, and cloud virtual machines. The authors reveal significant variations in performance metrics, with cold-start latency differing by up to 110 times and projected worker-hours varying by 1.8 times for one million 150-step trajectories. These findings highlight the necessity of integrating infrastructure optimization into the training process of coding-agent RL systems to enhance overall efficiency and scalability.
Up to 110x differences in cold-start latency reveal that execution infrastructure can dramatically impact the efficiency of coding-agent reinforcement learning.
Coding-agent reinforcement learning treats execution infrastructure as a background implementation detail, despite relying on large numbers of interactive software rollouts. This is a missed opportunity: measuring infrastructure overhead can reveal practical efficiency gains for RL post-training, where small per-rollout savings compound at scale. We present a comparative study of four execution substrates: single containers, hosted sandboxes, Kubernetes-orchestrated containers, and cloud virtual machines. We find up to $110\times$ variation in cold-start latency and a $1.8\times$ spread in projected worker-hours for one million 150-step trajectories. Our results suggest that future coding-agent RL systems should optimize execution substrates as part of the training system itself, not merely as deployment plumbing.
