Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper evaluates the realism of LLM-generated reactions to Spanish online news by comparing them to real audience replies from the Hatemedia dataset across hate speech, sentiment, and semantic alignment. The study found that off-the-shelf LLMs poorly replicate real audience reactions, particularly in underproducing hate speech and exhibiting sentiment biases. Fine-tuning improves fidelity, with Qwen3 providing the most balanced approximation, but even fine-tuned models struggle to fully capture the distributional properties of real public discourse.
Off-the-shelf LLMs are poor proxies for online social behavior, failing to capture the nuances of hate speech, sentiment, and semantic alignment found in real audience reactions to online news, even with fine-tuning.
LLM-powered social agents are increasingly used to simulate online social behavior, yet their realism remains difficult to validate. Existing work has largely relied on general-purpose benchmarks, while less attention has been paid to short, reactive discourse such as audience replies to online news. In this paper, we evaluate whether LLM-generated reactions to Spanish online news reproduce measurable properties of real audience discourse. Using the Hatemedia dataset, we pair 5,631 news items with 58,555 real audience reactions, and generate a matched synthetic dataset using five LLMs under a shared experimental setting. We compare real and synthetic reactions across three dimensions: hate speech, sentiment, and semantic alignment, considering both off-the-shelf and fine-tuned generation. Results show that off-the-shelf models are poor proxies for real audience reactions: they strongly underproduce hate speech, introduce model-specific sentiment biases, and remain distributionally distant from human replies. Fine-tuning improves fidelity unevenly. Qwen3 provides the most balanced approximation, while Mistral7B achieves the strongest sentiment and semantic alignment but overshoots hate prevalence. Plausible synthetic replies do not necessarily reproduce the distributional properties of public discourse.