Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study employs an agent-based model with 240 learning agents to investigate the shortcomings of content moderation systems that rely on aggregate accuracy metrics. It reveals that while overall system performance appears stable, harmful consequences disproportionately affect users who connect disparate communities—termed "bridge users." The findings highlight that governance losses, which account for the unique errors impacting these users, can more than double in the presence of noisy classifiers, underscoring the inadequacy of aggregate metrics in ensuring fair moderation outcomes.
Aggregate accuracy metrics can mask significant harm to vulnerable "bridge users" in content moderation systems, leading to a governance loss that more than doubles under noise.
A content-moderation system can score well on every standard accuracy metric and still cause real harm, if its mistakes fall on the few users who connect otherwise separate communities. We show this in an agent-based model where N=240 learning agents on a community-structured network each post harmless, productive, or dangerous content, and a regulator removes or penalizes whatever a noisy classifier flags. Overall usefulness barely moves as the noise changes (one-way ANOVA, p=0.96): by aggregate measures, nothing looks wrong. The damage instead concentrates on these bridge users, whose useful posts are wrongly suppressed and whose dangerous posts are wrongly spared. A governance loss (L_gov) that prices these two mistakes separately from the cost of enforcement more than doubles under false-positive-heavy noise. Aggregate accuracy hides who is harmed, and the cheap quantity to audit is how many connections a user has (degree), a near-perfect proxy for the betweenness that defines a bridge (r=0.96).