Search papers, labs, and topics across Lattice.
This paper addresses the challenge of compositional generalization in robotic manipulation by introducing a diagnostic framework that identifies and quantifies instruction factor bias in pretrained policies. By analyzing six foundational policies, the authors establish a consistent hierarchy of factor dominance, revealing that color is the most relied upon factor while verbs and size are frequently under-grounded. The proposed bias-aware data collection strategy effectively reallocates resources towards these under-grounded factors, leading to improved sample efficiency and generalizability in policy learning.
Fine-tuned robotic policies can be biased towards certain instruction factors, but a new bias-aware data strategy can significantly enhance their performance with fewer demonstrations.
Compositional generalization is essential for robot to follow diverse instructions. However, pretrained policies are known to take shortcuts, deferring to salient cues rather than grounding language. We introduce a diagnostic framework that localizes this failure to individual \textit{instruction factors}, \textit{e.g.,} reusable semantic components such as color, verb, object, size, and spatial attribute. Our framework formalizes instruction factor bias, the tendency of fine-tuned policies to over-rely on dominant factors as shortcuts, and quantifies it through two metrics: Factor Dominance Rate (FDR), capturing pairwise bias between factors, and Factor Dominance Hierarchy (FDH), aggregating these into a global ranking. Evaluation on six foundation policies reveals broadly consistent ordering, \textit{i.e.}, color $\geq$ object $\geq$ spatial $\geq$ verb $\geq$ size, with color dominant, and verb and size most under-grounded. We further show the diagnosis is actionable: a bias-aware data collection strategy that reallocates a fixed budget toward under-grounded factors outperforms baselines in simulation and on a real robot using half the demonstrations, thereby enabling more sample-efficient and generalizable policy learning.
NVIDIA