Search papers, labs, and topics across Lattice.
This study analyzes 19,146 collaborative notes generated by an LLM and refined through 211,850 instances of human feedback within X's Community Notes system. The findings reveal that human suggestions, particularly those challenging previous claims, significantly enhance the helpfulness of AI-generated notes, although these collaborative notes are still less frequently displayed than their human-only or AI-only counterparts. The research highlights the complementary role of AI in moderating misinformation, suggesting that while collaborative notes improve through human input, limited participation remains a critical barrier to their visibility and effectiveness.
Human feedback can transform AI-generated notes into more helpful content, but these collaborative efforts still struggle for visibility compared to traditional notes.
Community Notes, a bridging-based crowd-sourced fact-checking system, has emerged as a new mechanism for moderating misleading information on social media and has been adopted by major platforms including X, Facebook, Instagram, Threads, and TikTok. Since its introduction, there has been an open question about what role AI could play in scaling and optimizing the system. Recently, X extended its Community Notes system by introducing Collaborative Notes: notes initially drafted by an LLM and iteratively refined based on feedback from human contributors. In this work, we systematically analyze the complete corpus of 19,146 collaborative notes and 211,850 instances of human feedback. First, we develop a taxonomy of human suggestions for improving AI-generated note drafts and find that suggestions involving factual corrections and additional context are most likely to be incorporated, while subjective policy judgments rarely are. Second, we examine changes in helpfulness across versions of collaborative notes and find that human feedback leads to more helpful notes, with the greatest impact coming from suggestions that challenge the main claim in the previous draft, particularly when submitted by more active contributors. Finally, we find that although collaborative notes improve through human feedback, they reach helpful status and are shown on the platform at lower rates than human-only or AI-only notes, with limited human participation emerging as a key bottleneck. Nevertheless, rather than serving as a weaker substitute, collaborative notes tend to play a complementary role, predominantly targeting posts that do not attract human-only or AI-only notes. Our analysis provides an initial description of efforts to use AI to improve crowdsourced content moderation in a real-world moderation system and outlines pathways for future improvements to such features.
UW