Search papers, labs, and topics across Lattice.
This paper introduces a Decision Quality Evaluation Framework implemented at Pinterest for evaluating content moderation decisions made by both human agents and LLMs. The framework leverages a high-trust Golden Set (GDS) curated by subject matter experts as a ground truth benchmark and employs an intelligent sampling pipeline using propensity scores to expand dataset coverage efficiently. Results demonstrate the framework's utility in benchmarking LLM agent performance, optimizing prompts, managing policy evolution, and validating policy content prevalence metrics, enabling a data-driven approach to content safety.
Pinterest's new framework lets you quantitatively benchmark content moderation LLMs against human experts, revealing cost-performance trade-offs that were previously hidden.
Online platforms require robust systems to enforce content safety policies at scale. A critical component of these systems is the ability to evaluate the quality of moderation decisions made by both human agents and Large Language Models (LLMs). However, this evaluation is challenging due to the inherent trade-offs between cost, scale, and trustworthiness, along with the complexity of evolving policies. To address this, we present a comprehensive Decision Quality Evaluation Framework developed and deployed at Pinterest. The framework is centered on a high-trust Golden Set (GDS) curated by subject matter experts (SMEs), which serves as a ground truth benchmark. We introduce an automated intelligent sampling pipeline that uses propensity scores to efficiently expand dataset coverage. We demonstrate the framework's practical application in several key areas: benchmarking the cost-performance trade-offs of various LLM agents, establishing a rigorous methodology for data-driven prompt optimization, managing complex policy evolution, and ensuring the integrity of policy content prevalence metrics via continuous validation. The framework enables a shift from subjective assessments to a data-driven and quantitative practice for managing content safety systems.
