Search papers, labs, and topics across Lattice.
This paper introduces EmpiriGraph-Psy, a novel dataset and pipeline designed to extract variable-centered empirical relation graphs from psychology abstracts, addressing a significant gap in existing benchmarks that primarily focus on fields like computer science. By employing a staged graph-construction pipeline that separates various extraction tasks, the authors demonstrate a substantial improvement in performance, achieving a macro-F1 score of 0.74 with frontier and open-weight LLMs. The findings underscore the challenges in extracting complex empirical relations and concept hierarchies, particularly in higher-order claims within psychological research.
Extracting empirical relations from psychology abstracts reveals that even advanced LLMs struggle with higher-order claims, highlighting a critical gap in current AI capabilities.
Existing scientific relation extraction benchmarks mainly target domains such as computer science, where entities are tasks, methods, datasets, materials, or metrics. This leaves a gap in variable-oriented empirical fields such as psychology, where findings are expressed as relations among constructs, measurements, interventions, and outcomes. We introduce variable-centered empirical graph extraction, the task of mapping scientific abstracts to typed graphs whose nodes are normalized variables and whose edges represent empirical and hierarchical relations. To support this task, we construct EmpiriGraph-Psy, a benchmark of 210 psychology abstracts annotated by domain-trained annotators with normalized variables, concept hierarchies, empirical relation types, and validation states. We evaluate frontier and open-weight LLMs using both direct extraction and a staged graph-construction pipeline that separates variable extraction, normalization, hierarchy construction, evidence selection, relation extraction, and edge validation. The staged pipeline substantially outperforms direct extraction, with the best configuration achieving a macro-F1 of 0.74. Error analysis shows that moderation relations and concept hierarchies remain the most challenging cases, highlighting the difficulty of extracting higher-order empirical claims and implicit abstraction structure from scientific abstracts.
