Search papers, labs, and topics across Lattice.
This paper introduces KNOWSIM, a novel evaluation framework that utilizes a user simulator with explicit knowledge states to assess information calibration in Large Language Models (LLMs). By modeling user knowledge as a graph of Information Units and employing learning theory to evolve these states, KNOWSIM computes metrics that capture Knowledge Gain, Delivery Calibration, and Cognitive Overload. Validation against 705 human-AI interactions shows that KNOWSIM's rankings align closely with human judgments and reveals that LLM performance varies significantly with user knowledge levels, highlighting previously unnoticed aptitude-treatment interactions.
KNOWSIM uncovers that LLM performance is not uniform and shifts dramatically based on user knowledge levels, challenging traditional evaluation methods.
To effectively collaborate with users on knowledge-intensive tasks, Large Language Models (LLMs) must perform information calibration: matching content to a user's evolving understanding and cognitive capacity. Yet user simulators used to evaluate and train LLMs do not explicitly model user knowledge so they neither produce realistic interactions across knowledge levels nor reflect how interactions unfold as that knowledge evolves. To close this gap, we introduce KNOWSIM, an evaluation framework built around a user simulator that maintains explicit knowledge states, represented as a graph of Information Units with prerequisite relationships, that evolve under update rules grounded in learning theory. KNOWSIM computes three metrics (Knowledge Gain, Delivery Calibration, Cognitive Overload) directly from the knowledge state trajectory, reflecting key mechanistic aspects of information calibration. We validate KNOWSIM against 705 human-AI sessions across two domains, stratified by knowledge level: its rankings align significantly with human judgments (73-74% sign agreement), outperforming three baseline simulators. Applied to 9 LLMs, KNOWSIM reveals that the best model shifts by user knowledge level, revealing aptitude-treatment interactions invisible to standard evaluation.
Microsoft Research