Search papers, labs, and topics across Lattice.
This study introduces a human-in-the-loop (HITL) methodology for large-scale toxicity annotation on GitHub, addressing the challenges of implicit and context-dependent toxicity in open source discussions. By leveraging a local LLM for initial toxicity predictions and a Random Forest validator to identify conversations needing human review, the approach achieves high accuracy while minimizing annotation costs. The resulting dataset, derived from over 124,000 GitHub interactions, not only validates previous findings but also uncovers new insights into the dynamics of toxicity in diverse open source projects.
A novel HITL approach reveals that toxicity dynamics in open source projects are more complex than previously understood, with significant implications for contributor engagement.
Toxic interactions in open source discussions can alienate contributors and threaten project sustainability, yet prior empirical studies of GitHub toxicity have been limited in scale, raising questions about their generalizability. Scaling up is difficult because toxicity on GitHub is often implicit and context-dependent, making both fully manual annotation and LLM-based labeling unreliable. We present a human-in-the-loop (HITL) annotation methodology that makes large-scale, domain-calibrated toxicity labeling practical. A single call to a small, local LLM produces both a toxicity prediction and a set of interpretable event category scores. A lightweight Random Forest validator then uses those scores to flag the small subset of conversations most likely to be mislabeled, directing human review only where it is needed. The validator outperforms confidence-based and multi-LLM baselines while adding low annotation cost. We apply this pipeline to over 124,000 GitHub issue and pull request conversations. Using the resulting dataset, we evaluate key findings from prior small-scale research, confirming some and qualifying others, and present new insights into the prevalence, characteristics, and dynamics of toxicity across diverse open source projects.
