Search papers, labs, and topics across Lattice.
This paper investigates the representation and usage of African American Vernacular English (AAVE) in large language models (LLMs) by comparing model-generated text to human usage patterns derived from the Corpus of Regional African American Language and TwitterAAE. The authors prompted three LLMs to generate text in AAVE and analyzed the frequency and context of AAVE grammatical features. The study reveals that LLMs often underuse and misuse AAVE grammatical features and, furthermore, replicate harmful stereotypes about African Americans.
LLMs don't just misunderstand AAVE, they also perpetuate harmful stereotypes when attempting to use it.
In AI, most evaluations of natural language understanding tasks are conducted in standardized dialects such as Standard American English (SAE). In this work, we investigate how accurately large language models (LLMs) represent African American Vernacular English (AAVE). We analyze three LLMs to compare their usage of AAVE to the usage of humans who natively speak AAVE. We first analyzed interviews from the Corpus of Regional African American Language and TwitterAAE to identify the typical contexts where people use AAVE grammatical features such as ain't. We then prompted the LLMs to produce text in AAVE and compared the model-generated text to human usage patterns. We find that, in many cases, there are substantial differences between AAVE usage in LLMs and humans: LLMs usually underuse and misuse grammatical features characteristic of AAVE. Furthermore, through sentiment analysis and manual inspection, we found that the models replicated stereotypes about African Americans. These results highlight the need for more diversity in training data and the incorporation of fairness methods to mitigate the perpetuation of stereotypes.
