Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study evaluates the spatial reasoning capabilities of large language models (LLMs) by testing their ability to generate ASCII art representations of planar graphs from edge lists. By assessing 91 models on 199 non-isomorphic connected planar graphs, the authors reveal that the edge count is a more significant predictor of difficulty than node count, a novel insight not previously documented in LLM graph benchmarks. The findings suggest that current evaluation metrics may overlook critical aspects of spatial reasoning in LLMs, highlighting the need for more nuanced assessments.
Edge count, not node count, emerges as the key predictor of difficulty in LLM spatial reasoning tasks, challenging existing benchmarks.
PlanarBench tests whether LLMs can draw planar graphs as ASCII art given only an edge list -- a spatial reasoning task that resists memorization because edge order, edge orientation, and node labels are all permutable. We evaluate 91 models on the 199 simplest non-isomorphic connected planar graphs (2 - 7 vertices). Edge count is the dominant difficulty predictor ($r = -0.85$) -- a finding not reported in prior LLM graph benchmarks, which use only node count as the difficulty axis.