Search papers, labs, and topics across Lattice.
This study conducts a zero-shot evaluation of three leading large language models鈥擟laude, ChatGPT, and Gemini鈥攐n a fine-grained emotion classification task involving 13 distinct emotional categories. The evaluation utilized a stratified sample of 1,000 sentences from the boltuix/emotions dataset and revealed that while Gemini outperformed the others with a 39.9% accuracy, all models struggled with emotions like love, confusion, and shame. The results indicate a shared ceiling effect among the models and highlight significant biases, particularly in Claude, underscoring the limitations of current AI systems in accurately classifying nuanced human emotions.
All leading LLMs struggle with fine-grained emotion classification, revealing a surprising ceiling effect in zero-shot performance.
Emotion recognition in natural language is a foundational challenge in affective computing, with critical implications for human-computer interaction, mental health support, and conversational AI. This paper presents a rigorous, unified zero-shot evaluation of three leading commercial large language models: Claude (claude-sonnet-4-6), ChatGPT (GPT-5.4), and Gemini (gemini-2.5-flash). The models were queried through their respective production APIs as of April 2026 on a fine-grained 13-class emotion classification task. Using a stratified 1,000-sentence sample from the boltuix/emotions dataset, which comprises 131,306 sentences across 13 categories, a single uniform prompt with no exemplars was applied identically across all models. Gemini achieves the highest accuracy (39.9%) and macro-F1 score (0.363), followed by GPT-5.4 (38.8%, macro-F1 = 0.291) and Claude (38.0%, macro-F1 = 0.159). All models excel on sarcasm and desire while consistently failing on love, confusion, and shame. McNemar tests reveal no statistically significant pairwise differences (p > 0.10), suggesting convergence at a shared zero-shot ceiling. Claude's markedly lower macro-F1 score exposes a class-imbalance prediction bias. These findings highlight the current limitations of frontier AI systems in zero-shot fine-grained emotion classification.
