Search papers, labs, and topics across Lattice.
This paper introduces a novel benchmark using the publication histories of 217 AI researchers to evaluate if LLMs can simulate human cognition beyond behavioral imitation. The benchmark employs a cross-domain, temporal-shift generalization setting to test for genuine cognitive pattern transfer rather than superficial imitation. Results show that current LLMs struggle to simulate individual-level cognitive consistency as measured by a proposed multidimensional cognitive alignment metric, even with enhancement techniques.
LLMs may ace the test, but they're failing to think like us: a new benchmark reveals their struggle to simulate individual cognitive consistency across research domains and time.
An essential problem in artificial intelligence is whether LLMs can simulate human cognition or merely imitate surface-level behaviors, while existing datasets suffer from either synthetic reasoning traces or population-level aggregation, failing to capture authentic individual cognitive patterns. We introduce a benchmark grounded in the longitudinal research trajectories of 217 researchers across diverse domains of artificial intelligence, where each author's scientific publications serve as an externalized representation of their cognitive processes. To distinguish whether LLMs transfer cognitive patterns or merely imitate behaviors, our benchmark deliberately employs a cross-domain, temporal-shift generalization setting. A multidimensional cognitive alignment metric is further proposed to assess individual-level cognitive consistency. Through systematic evaluation of state-of-the-art LLMs and various enhancement techniques, we provide a first-stage empirical study on the questions: (1) How well do current LLMs simulate human cognition? and (2) How far can existing techniques enhance these capabilities?
