Search papers, labs, and topics across Lattice.
This paper introduces the "Individual Turing Test" to evaluate LLMs' ability to simulate specific individuals using a volunteer's decade-long private messaging history. The study compares fine-tuning, RAG, memory-based methods, and their hybrids, assessing whether acquaintances can distinguish the volunteer's responses from those generated by the models. Results indicate that while current LLMs fail the Individual Turing Test against acquaintances, they perform better against strangers, and that fine-tuning excels at replicating language style while RAG/memory-based methods are better at capturing personal opinions.
LLMs can mimic your style, but your friends can still tell it's not really you, especially when it comes to your opinions.
Large Language Models (LLMs) have demonstrated remarkable human-like capabilities, yet their ability to replicate a specific individual remains under-explored. This paper presents a case study to investigate LLM-based individual simulation with a volunteer-contributed archive of private messaging history spanning over ten years. Based on the messaging data, we propose the "Individual Turing Test" to evaluate whether acquaintances of the volunteer can correctly identify which response in a multi-candidate pool most plausibly comes from the volunteer. We investigate prevalent LLM-based individual simulation approaches including: fine-tuning, retrieval-augmented generation (RAG), memory-based approach, and hybrid methods that integrate fine-tuning and RAG or memory. Empirical results show that current LLM-based simulation methods do not pass the Individual Turing Test, but they perform substantially better when the same test is conducted on strangers to the target individual. Additionally, while fine-tuning improves the simulation in daily chats representing the language style of the individual, retrieval-augmented and memory-based approaches demonstrate stronger performance on questions involving personal opinions and preferences. These findings reveal a fundamental trade-off between parametric and non-parametric approaches to individual simulation with LLMs when given a longitudinal context.
Microsoft Research