Search papers, labs, and topics across Lattice.
This paper introduces GraphDecisionBench, a novel benchmark designed to rigorously evaluate System One models, like Jev, on graph understanding and decision-making tasks by employing structural task profiles, matched graph-text input contrasts, and heuristic-proposal controls. The evaluation reveals that while Jev exhibits accurate adjacency recognition, it struggles with broader structural correctness, and joint graph-text inputs do not consistently enhance performance, highlighting limitations in current choice-based models. The benchmark's design, including task contracts and candidate interfaces, facilitates a nuanced comparison between native selectors and LLM adapters on graph-related decision problems.
Choice-based models like Jev can accurately identify graph adjacencies but fail to grasp broader structural correctness, even with joint graph-text inputs.
Large language models (LLMs) are increasingly explored for graph understanding and decision-making, while System One models such as Jev select directly from supplied options. However, the capabilities of System One models on graph-related tasks remain unclear. We introduce GraphDecide, a model-independent benchmark that combines structural task profiles, matched graph-text input contrasts and heuristic-proposal controls to diagnose graph decision performance. We evaluate Jev and related choice-based models alongside language-model baselines, covering fourteen model-interface configurations. Jev's results illustrate the benchmark's central distinctions: accurate adjacency recognition does not guarantee broader structural correctness, joint graph-text input does not consistently improve prediction, and feasible construction does not establish high solution quality. Its task contracts, candidate interfaces and scoring rules support comparison across native selectors and language-model adapters. Code and aggregate results are available at https://github.com/VictorYXL/JevGraphBench.
Microsoft Research