Search papers, labs, and topics across Lattice.
To expose the latent failure modes of Vision-Language-Action (VLA) models, the authors release RoboSPA, a diagnostic benchmark and 527K-trajectory dataset spanning 56 base manipulation tasks across five systematically scaled difficulty tiers. Unlike conventional benchmarks that rely solely on binary success rates in static environments, this work isolates performance across fine-grained spatial reasoning and long-horizon procedural planning. Empirical evaluations reveal that leading VLA architectures degrade sharply when subjected to spatial ambiguity, precise low-level execution constraints, and memory-intensive multi-step reasoning.
Scaling task complexity across 280 standardized manipulation variants reveals that today's leading VLA models fundamentally collapse under fine-grained spatial ambiguity and memory-intensive sequential planning.
Vision-Language-Action (VLA) models have shown promising progress in language-conditioned robotic manipulation. However, existing datasets and benchmarks mainly evaluate task completion under predefined settings, offering limited insight into model reasoning under increasing spatial and procedural complexity. We introduce \textbf{RoboSPA} (\textbf{Robo}t \textbf{S}patial-\textbf{P}rocedural \textbf{A}ssessment), a large-scale robotic manipulation dataset and benchmark for diagnosing embodied reasoning in VLA models. \texttt{RoboSPA} focuses on two core dimensions, Fine-Grained Spatial Reasoning and Long-Horizon Procedural Planning, covering 10 task categories and 56 base tasks. Each task is instantiated across five difficulty levels, yielding 280 variants with increasing spatial ambiguity and procedural complexity. We collect 527K trajectories across multiple embodiments and diverse scenes. Beyond binary success rate, \texttt{RoboSPA} introduces diagnostic metrics for more detailed evaluation. Experiments on representative VLA models show that current systems still struggle with complex spatial relations, precise low-level execution, and memory-intensive planning. These results establish \texttt{RoboSPA} as a challenging diagnostic benchmark for developing more capable, reliable, and generalizable embodied agents. Our data and code are available at https://github.com/fanzhenxuan/RoboSPA.
