Search papers, labs, and topics across Lattice.
This paper analyzes the unique characteristics of issue resolution in AI/ML systems through a qualitative study of 100 issue reports and pull requests from prominent frameworks like TensorFlow and scikit-learn. The research highlights the distinct challenges posed by stochastic behavior, iterative experimentation, and the need for coordination across diverse artifacts, which traditional software maintenance frameworks fail to address. Key findings reveal that effective issue resolution in AI/ML requires tailored frameworks that accommodate these complexities to improve reproducibility and adaptive verification processes.
AI/ML issue resolution is fundamentally different from traditional software maintenance, requiring new frameworks to tackle unique challenges like stochastic behavior and artifact coordination.
We advocate for AI/ML issue resolution frameworks tailored to maintenance workflows and the nature of modern AI/ML systems. Existing issue resolution frameworks largely emerged for traditional software maintenance practices and do not explicitly account for characteristics common in AI/ML systems, such as stochastic behavior, experimentation-driven workflows, and heterogeneous artifacts beyond source code. To identify the unique characteristics of issue resolution in AI/ML systems and motivate the need for tailored frameworks, we conducted a qualitative study of issue resolution workflows documented in 100 issue reports and pull requests across four widely used AI/ML systems: TensorFlow, scikit-learn, MLflow, and AutoGPT. Our findings suggest that issue resolution in AI/ML systems involves: recurring AI/ML-related activities that span multiple resolution stages; iterative experimentation and adaptive verification; and coordinated changes across artifacts such as datasets, prompts, and model configurations. We also observed challenges related to reproducibility, nondeterministic behavior, and artifact coordination. Building on these findings, we present a vision for AI/ML issue resolution frameworks and discuss research directions and tooling support needed to realize this vision.
