Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper explores the emerging "policy-as-prompt" approach in content moderation, where community guidelines are transformed into natural language prompts for large language models (LLMs) to assist in moderation tasks. The authors argue that while this method can alleviate some burdens of traditional moderation, it introduces significant risks and harms that must be addressed to ensure effective community governance. Ultimately, they conclude that relying solely on prompt formulation is insufficient for meaningful governance, highlighting the need for more robust strategies.
Relying on LLMs for content moderation may introduce more risks than it mitigates, challenging the effectiveness of community governance.
Content moderation practices and governance paradigms are changing rapidly, as fewer human moderators are deployed as `experts' by social media companies in a centralized manner. Instead, the companies are focusing more on community approaches, relying on volunteers to provide accurate information and make correct decisions. In decentralized moderation, communities have always relied on volunteers, updated community guidelines, and internal discussions thereof. For both content moderation paradigms, Artificial Intelligence (AI) seems like it could help ease moderation burdens of time, mental health, and accuracy. One possible way to operationalize AI in content moderation is a `policy-as-prompt'' approach, where the policy is formulated as a natural-language prompt and then passed to a large language model (LLM). This model then aids in moderation tasks. In this paper, we briefly lay out the technical and governance properties of this approach, and argue that its limitations lead to specific risks and harms that have to be addressed. Towards alleviating them, we lay out multiple considerations towards more effective prompt governance, but ultimately find that writing prompts alone is not appropriate for ensuring meaningful community governance.