Search papers, labs, and topics across Lattice.
This paper explores the application of Centaur, a foundation model trained on extensive psychological data, to predict program comprehension in software engineering. By evaluating its performance against human response patterns across nine studies, the authors find that Centaur aligns more closely with human behavior than its base model, Llama 3.1, and demonstrates a reduced reliance on prior trial information while benefiting from task-related context. These results highlight the potential of leveraging cognitive regularities from behavioral data to enhance our understanding of developer behavior in complex coding tasks.
Centaur's predictions align more closely with human responses in program comprehension than traditional models, revealing the power of cognitive insights in software engineering.
Software engineering depends on the ability of developers to understand code, yet predicting how they do so remains an open challenge despite decades of research. Existing approaches rely either on simplified proxy measures that limit accuracy or on non-trivial measurements requiring elaborate experimental setups that are difficult to scale and apply in practice. In contrast, recent work in psychology suggests an alternative perspective: Instead of modeling task-specific phenomena directly, human behavior can be captured through cognitive regularities learned from large-scale behavioral data. This idea treats complex human behavior as the observable outcome of underlying cognitive processes that manifest consistently across tasks and domains. In this paper, we explore this perspective in the context of program comprehension. We evaluate Centaur, a foundation model trained on 160 general psychological experiments, on 9 previously published program-comprehension studies. We assess how well its predicted response distributions align with human response data and compare Centaur's performance to its base model, Llama 3.1. To better understand the source of its performance, we conduct ablation studies to isolate the contribution of different sources of information, such as the code artifacts, task-related context, and prior trials and participant responses. In a nutshell, we find that Centaur more closely aligns with human response patterns than its base model, is significantly less reliant on information from prior trials and responses, and benefits more from task-related information. These findings suggest that behavioral patterns learned from general psychological data can transfer to complex software engineering tasks such as program comprehension. More broadly, they point toward foundation models of human cognition as a basis for modeling developer behavior in software engineering.
