Search papers, labs, and topics across Lattice.
This study systematically evaluates the impact of trajectory data quality and quantity on the performance of code agents fine-tuned using LoRA on the Qwen2.5-Coder-7B-Instruct model. By introducing a two-axis quality scoring framework鈥擡fficiency and Style鈥攁nd conducting 16 controlled experiments, the authors reveal a scale-dependent trade-off where increasing dataset size significantly reduces cross-entropy loss, particularly at smaller scales. The findings highlight the importance of trajectory-level quality scoring as a critical factor in the supervised fine-tuning of code agents, especially when traditional resolve rates are not feasible to measure.
A scale-dependent quality-quantity trade-off reveals that doubling trajectory data can significantly enhance model performance, but only up to a point.
Supervised fine-tuning (SFT) of open-weight LLMs on expert agent trajectories has emerged as a prominent approach to building capable code agents without reliance on proprietary models. A central yet underexplored question is how trajectory quality and quantity jointly shape model performance. We present a systematic empirical study of trajectory data filtering for LoRA fine-tuning of Qwen2.5-Coder-7B-Instruct on the SWE-trajectory dataset (67,074 trajectories, of which 32,161 are resolved). We propose a two-axis quality scoring framework -- Efficiency and Style -- and evaluate it through 16 controlled experiments spanning strategy, scale, and ablation analyses. Since 7B-scale models attain near-zero SWE-bench resolve rates, we adopt cross-entropy (CE) loss on held-out trajectories as the primary metric, validated via first-action generation: CE loss and ROUGE-L are perfectly rank-correlated (Spearman $蟻$ = -1.00), with limited-sample evidence supporting but not conclusively establishing this proxy. Our results reveal a scale-dependent quality-quantity trade-off: at small scales, doubling the dataset (500 to 1,000) yields ~12.7% CE-loss reduction whereas the TopQ-Random gap stays <1% (Mann-Whitney p > 0.10); at 2,000 trajectories this same gap widens to 3.6% (p = 0.016). Ablation further identifies error-retry rate as the dominant sub-dimension, performing comparably to the full composite ($螖$ < 0.2%). Together, these findings establish trajectory-level quality scoring as a viable but scale-sensitive lever for code-agent SFT and offer a proxy-validated evaluation protocol for the regime where end-to-end resolve rate is statistically infeasible.
