Search papers, labs, and topics across Lattice.
The paper introduces SWE-Future, a novel method for synthesizing coding tasks by leveraging forecast-conditioned data synthesis to predict future software engineering needs based on historical repository evidence. This approach addresses the limitations of traditional benchmarks that rely on historical data, which can lead to model biases and overfitting. In an evaluation across 80 repositories, SWE-Future achieved a 58.1% relevance rate for future work, demonstrating its effectiveness in generating realistic coding tasks aligned with actual repository evolution.
Forecasting future coding tasks can yield a dataset that is 58.1% relevant to real-world software engineering needs, sidestepping the pitfalls of historical data replay.
Realistic coding-agent benchmarks often replay public GitHub issues and pull requests, making them vulnerable to overlap with model pretraining, fine-tuning, synthetic-data generation, or benchmark-driven model selection. Fully synthetic tasks avoid direct historical replay, but can drift away from real repository needs. We propose SWE-Future, a forecast-conditioned data synthesis method for future-oriented coding tasks. Given a forecast snapshot at time $T_0$, the method uses only pre-$T_0$ repository evidence to forecast future feature implementation/enhancement, bugfix, and refactor task families. We first validate this forecasting step retrospectively: after forecasts are fixed, later pull requests are used only to measure whether the predicted task families match future repository work. In an 80-repository study, the forecaster achieves 58.1\% future-work relevance under the main semantic matching metric. We then use validated forecast families as conditioning signals to synthesize a 200-task coding-agent dataset across 61 repositories from a task-generation snapshot, rather than replaying the later pull requests used for validation. SWE-Future shows that repository-evolution forecasts can guide realistic, future-oriented coding-task synthesis while reducing direct dependence on historical pull-request replay.
