Search papers, labs, and topics across Lattice.
The paper introduces ContentBench, a benchmark suite for evaluating the ability of low-cost LLMs to perform interpretive coding tasks, specifically focusing on classifying social-media-style posts about academic research. The benchmark uses a consensus-based labeling approach, where labels are assigned only when three state-of-the-art reasoning models agree unanimously, followed by a quality-control audit. Results from the first track, ContentBench-ResearchTalk v1.0, show that top low-cost LLMs achieve high agreement (97-99%) with the jury labels, suggesting they can potentially replace human coders for large-scale interpretive coding tasks, while smaller open-weight models struggle with nuanced categories like sarcasm.
Forget expensive human coders: some low-cost LLMs can now achieve near-perfect agreement on complex interpretive coding tasks, but sarcasm remains a challenge for smaller models.
Can low-cost large language models (LLMs) take over the interpretive coding work that still anchors much of empirical content analysis? This paper introduces ContentBench, a public benchmark suite that helps answer this replacement question by tracking how much agreement low-cost LLMs achieve and what they cost on the same interpretive coding tasks. The suite uses versioned tracks that invite researchers to contribute new benchmark datasets. I report results from the first track, ContentBench-ResearchTalk v1.0: 1,000 synthetic, social-media-style posts about academic research labeled into five categories spanning praise, critique, sarcasm, questions, and procedural remarks. Reference labels are assigned only when three state-of-the-art reasoning models (GPT-5, Gemini 2.5 Pro, and Claude Opus 4.1) agree unanimously, and all final labels are checked by the author as a quality-control audit. Among the 59 evaluated models, the best low-cost LLMs reach roughly 97-99% agreement with these jury labels, far above GPT-3.5 Turbo, the model behind early ChatGPT and the initial wave of LLM-based text annotation. Several top models can code 50,000 posts for only a few dollars, pushing large-scale interpretive coding from a labor bottleneck toward questions of validation, reporting, and governance. At the same time, small open-weight models that run locally still struggle on sarcasm-heavy items (for example, Llama 3.2 3B reaches only 4% agreement on hard-sarcasm). ContentBench is released with data, documentation, and an interactive quiz at contentbench.github.io to support comparable evaluations over time and to invite community extensions.
