Search papers, labs, and topics across Lattice.
This study investigates the potential of individual text corpora (ICs) derived from search histories to simulate personalized knowledge by comparing the performance of various large language models (LLMs) on knowledge assessment tasks. Among the models tested, only Qwen3-1.7B, after fine-tuning with Low-Rank Adaptation (LoRA), surpassed both participant responses and a German norm sample on publicly available knowledge items, though it struggled with non-public questions, indicating potential issues with training data contamination. The integration of ICs into a retrieval-augmented generation framework yielded significant match accuracies between LLM predictions and participant responses, revealing a detectable signal of individual knowledge despite low calibration in probability assignments.
Qwen3-1.7B can outperform human participants in knowledge assessments when fine-tuned on individual text corpora, but struggles with calibration and non-public questions.
This study examines whether individual text corpora (ICs) from search histories can be used to simulate individual knowledge. We collected ICs from 316 adults, who answered 36 multiple-choice knowledge items, and compared several large language models (LLMs) on this task, of which only Qwen3-1.7B proved viable. After task-specific fine-tuning via Low-Rank Adaptation (LoRA), Qwen3-1.7B outperformed both participants and a representative German norm sample on publicly available items. On non-public questions, however, the LLM performed worse than our participants, suggesting possible training data contamination for the public questions. When integrating ICs into retrieval-augmented generation to predict individual responses, LLM-participant Match accuracies significantly exceeded chance, which demonstrates a detectable individual knowledge signal. The probabilities assigned to the participants'answers were, however, low and far below the probability of correct answers, indicating poor calibration toward individual response patterns. Knowledge-gap prediction was sub-optimal, though it improved for corpora exceeding five million tokens. We discuss our entropy based evaluation benchmarks as calibration indices for individualized knowledge simulation.
