Search papers, labs, and topics across Lattice.
The paper introduces SWE-MIMIC-Bench, a dataset of issue trajectories extracted from GitHub discussions using a novel multi-LLM pipeline. This pipeline decomposes the task into granular steps, leveraging five closed-source LLM configurations for comment analysis, classification, and trajectory synthesis, going beyond simple summarization. The resulting dataset provides structured narratives of issue resolution, achieving a 91.7% success rate in extracting high-fidelity reasoning trajectories from 800 real-world GitHub issues.
Unlock expert developer reasoning: a new dataset distills complex GitHub issue discussions into structured trajectories, revealing the collaborative problem-solving process behind open-source software.
Resolution of complex post-production issues in large-scale open-source software (OSS) projects requires significant cognitive effort, as developers need to go through long, unstructured and fragmented issue discussion threads before that. In this paper, we present SWE-MIMIC-Bench, an issue trajectory dataset generated from raw GitHub discussions using an automated multi-LLM pipeline. Unlike simple summarization, this pipeline utilizes a group of closed-source LLMs to perform granular tasks: analyzing individual comments with awareness of externally-linked resources, classifying comment analyses into label-specific fields (e.g., root cause, solution plan, implementation progress), and synthesizing label-aware trajectories which capture a structured and coherent narrative of the entire discussion thread. Our pipeline uses five closed-source LLM configurations for distinct purposes: label classification, inline code block and external link summarization, comment analysis, label-specific field classification and trajectory synthesis. By generating concise and reliable trajectories from complex conversation threads, this system can assist developers and researchers of broader software engineering community to understand the experience-driven collaborative approach for issue diagnosis. Furthermore, the generated trajectories can be used to train modern LLM agents to think and act like an expert developer. We evaluated our system on 800 real-world GitHub issues drawn from the SWE-Bench-Pro, SWE-Bench-Multilingual and SWE-Bench-Verified dataset, achieving a 91.7% success rate in extracting 734 high-fidelity reasoning trajectories.
