Search papers, labs, and topics across Lattice.
This paper analyzes the NLP community's reliance on Google's Perspective API for toxicity measurement, highlighting issues of reproducibility, standardization around a proprietary tool, and lack of transparency in model updates. It argues that this dependence has created epistemic problems, including non-updatable benchmarks and irreproducible results. The authors advocate for the development of an independent, open, and reproducible toxicity measurement infrastructure to address these shortcomings.
The shutdown of Perspective API exposes a critical vulnerability in NLP research: over-reliance on opaque, proprietary tools for toxicity measurement, threatening the validity and reproducibility of past and future work.
The closure of Perspective API at the end of 2026 discards what has functioned as the de facto standard for automated toxicity measurement in NLP, CSS, and LLM evaluation research. We document the structural dependence that the communities built on this single proprietary tool and discuss how this dependence caused epistemic problems that have affected - and will likely continue to affect - collective research efforts. Perspective's model was periodically updated without versioning or disclosure, its annotation structure reflected a single corporate operationalisation of a contested concept, and its scores were used simultaneously as an evaluation target and an evaluation standard. Its closure leaves behind non-updatable benchmarks, irreproducible results, and ultimately a field at risk of perpetuating these issues by turning to closed-source LLMs. We use Perspective's announced termination as an opportunity to call for an independent, valid, adaptable, and reproducible toxicity and hate speech measurement infrastructure, with the technical and governance requirements outlined in this paper.
