Search papers, labs, and topics across Lattice.
This paper analyzes how well programming agents adhere to instructed plans within reason-act-observe loops, using 16,991 trajectories from SWE-agent across four LLMs on SWE-bench. The study finds that agents often deviate from provided plans, falling back on internalized workflows that can be incomplete or overfit, and that subpar plans can be detrimental to performance. Periodic plan reminders improve task success, while augmenting plans with early-stage phases that misalign with the model's internal strategy can degrade performance.
Turns out, LLM agents are surprisingly bad at following plans, often preferring their own internalized (and potentially flawed) workflows, and a bad plan is worse than no plan at all.
Agents aspire to eliminate the need for task-specific prompt crafting through autonomous reason-act-observe loops. Still, they are commonly instructed to follow a task-specific plan for guidance, e.g., to resolve software issues following phases for navigation, reproduction, patch, and validation. Unfortunately, it is unknown to what extent agents actually follow such instructed plans. Without such an analysis, determining the extent agents comply with a given plan, it is impossible to assess whether a solution was reached through correct strategic reasoning or through other means, e.g., data contamination or overfitting to a benchmark. This paper presents the first extensive, systematic analysis of plan compliance in programming agents, examining 16,991 trajectories from SWE-agent across four LLMs on SWE-bench Verified and SWE-bench Pro under eight plan variations. Without an explicit plan, agents fall back on workflows internalized during training, which are often incomplete, overfit, or inconsistently applied. Providing the standard plan improves issue resolution, and we observe that periodic plan reminders can mitigate plan violations and improve task success. A subpar plan hurts performance even more than no plan at all. Surprisingly, augmenting a plan with additional task-relevant phases in the early stage can degrade performance, particularly when these phases do not align with the model's internal problem-solving strategy. These findings highlight a research gap: fine-tuning paradigms that teach models to follow instructed plans, rather than encoding task-specific plans in them. This requires teaching models to reason and act adaptively, rather than memorizing workflows.
