Search papers, labs, and topics across Lattice.
This paper investigates biases in skin-toned emoji representations within both dedicated emoji embedding models (emoji2vec, emoji-sw2v) and large language models (Llama, Gemma, Qwen, Mistral). Through a multi-faceted analysis of semantic consistency, representational similarity, sentiment polarity, and core biases, the study reveals that while LLMs handle skin tone modifiers relatively well, specialized emoji models exhibit significant deficiencies and systemic disparities in sentiment and meaning across skin tones. The findings highlight the presence of latent biases in these models, emphasizing the need for auditing and mitigation strategies.
Widely-used emoji embedding models show surprisingly severe biases in how they represent different skin tones, even compared to LLMs.
Skin-toned emojis are crucial for fostering personal identity and social inclusion in online communication. As AI models, particularly Large Language Models (LLMs), increasingly mediate interactions on web platforms, the risk that these systems perpetuate societal biases through their representation of such symbols is a significant concern. This paper presents the first large-scale comparative study of bias in skin-toned emoji representations across two distinct model classes. We systematically evaluate dedicated emoji embedding models (emoji2vec, emoji-sw2v) against four modern LLMs (Llama, Gemma, Qwen, and Mistral). Our analysis first reveals a critical performance gap: while LLMs demonstrate robust support for skin tone modifiers, widely-used specialized emoji models exhibit severe deficiencies. More importantly, a multi-faceted investigation into semantic consistency, representational similarity, sentiment polarity, and core biases uncovers systemic disparities. We find evidence of skewed sentiment and inconsistent meanings associated with emojis across different skin tones, highlighting latent biases within these foundational models. Our findings underscore the urgent need for developers and platforms to audit and mitigate these representational harms, ensuring that AI's role on the web promotes genuine equity rather than reinforcing societal biases.
