Search papers, labs, and topics across Lattice.
The paper introduces GMP, a new benchmark designed to evaluate content moderation systems under conditions of co-occurring policy violations and dynamic moderation rules, addressing limitations of existing static benchmarks. GMP includes diverse scenarios with multiple policy violations within a single post and incorporates platform-specific guidelines that evolve across contexts. Experiments using GMP reveal that current AI systems, including LLMs, struggle to maintain consistent judgment when faced with unstable or context-dependent policies, highlighting a gap between performance on static benchmarks and real-world content moderation robustness.
Current content moderation benchmarks fail to capture the complexities of real-world scenarios, where posts often violate multiple policies simultaneously and moderation rules are constantly evolving, leading to inconsistent AI judgment.
Online content moderation is essential for maintaining a healthy digital environment, and reliance on AI for this task continues to grow. Consider a user comment using national stereotypes to insult a politician. This example illustrates two critical challenges in real-world scenarios: (1) Co-occurring Violations, where a single post violates multiple policies (e.g., prejudice and personal attacks); (2) Dynamic rules of moderation, where determination of a violation depends on platform-specific guidelines that evolve across contexts . The intersection of co-occurring harms and dynamically changing rules highlights a core limitation of current AI systems: although large language models (LLMs) are adept at following fixed guidelines, their judgment capabilities degrade when policies are unstable or context-dependent . In practice, such shortcomings lead to inconsistent moderation: either erroneously restricting legitimate expression or allowing harmful content to remain online . This raises a critical question for evaluation: Does high performance on existing static benchmarks truly guarantee robust generalization of AI judgment to real-world scenarios involving co-occurring violations and dynamically changing rules?
