Search papers, labs, and topics across Lattice.
This paper investigates the impact of training large language models (LLMs) on fictional text, specifically novels, compared to other text sources like news, Reddit, and Wikipedia. The authors analyze the behavior of BERT, an influential open-source model, to understand how it leverages the attributes of fiction learned during pre-training. They find that LLMs trained on novels exhibit unique characteristics and influence social responses in novel ways, highlighting the importance of considering computational training data in cultural analysis.
LLMs don't just learn from novels, they amplify the unique qualities of fictional discourse, suggesting that the genre of training data significantly shapes AI outputs.
Generative models, like the one in ChatGPT, are powered by their training data. The models are simply next-word predictors, based on patterns learned from vast amounts of pre-existing text. Since the first generation of GPT, it is striking that the most popular datasets have included substantial collections of novels. For the engineers and research scientists who build these models, there is a common belief that the language in fiction is rich enough to cover all manner of social and communicative phenomena, yet the belief has gone mostly unexamined. How does fiction shape the outputs of generative AI? Specifically, what are novels' effects relative to other forms of text, such as newspapers, Reddit, and Wikipedia? Since the 1970s, literature scholars such as Catherine Gallagher and James Phelan have developed robust and insightful accounts of how fiction operates as a form of discourse and language. Through our study of an influential open-source model (BERT), we find that LLMs leverage familiar attributes and affordances of fiction, while also fomenting new qualities and forms of social response. We argue that if contemporary culture is increasingly shaped by generative AI and machine learning, any analysis of today's various modes of cultural production must account for a relatively novel dimension: computational training data.
