Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces the Spatial Competence Benchmark (SCBench) to evaluate large models' ability to maintain consistent spatial representations and use them for planning. SCBench comprises three levels of tasks requiring executable outputs and deterministic verification. Evaluation of frontier models on SCBench reveals a decline in accuracy as task complexity increases, with performance gains saturating quickly with increased output tokens, suggesting limitations in global constraint satisfaction.
Frontier models struggle with global spatial reasoning, often generating locally plausible but globally inconsistent solutions, even with increased output token budgets.
Spatial competence is the quality of maintaining a consistent internal representation of an environment and using it to infer discrete structure and plan actions under constraints. Prevailing spatial evaluations for large models are limited to probing isolated primitives through 3D transformations or visual question answering. We introduce the Spatial Competence Benchmark (SCBench), spanning three hierarchical capability buckets whose tasks require executable outputs verified by deterministic checkers or simulator-based evaluators. On SCBench, three frontier models exhibit monotonically decreasing accuracy up the capability ladder. Sweeping output-token caps shows that accuracy gains concentrate at low budgets and saturate quickly, and failures are dominated by locally plausible geometry that breaks global constraints. We release the task generators, verifiers, and visualisation tooling.