Search papers, labs, and topics across Lattice.
This paper investigates the application of Metamorphic Testing (MT) to address the test oracle problem in Vision-Language-Action (VLA) robotic task controllers. They propose two metamorphic relation patterns and five metamorphic relations to evaluate the impact of input changes on the robot's trajectory. Results from experiments with five VLA models, two simulated robots, and four tasks demonstrate that MT effectively detects failures and offers a generalizable approach applicable across different VLA models, robots, and tasks without relying on specific test oracles.
Metamorphic Testing offers a surprisingly effective, oracle-free approach to uncovering failures in vision-language-action robots, even detecting subtle issues like uncompleted tasks.
Vision-Language-Action (VLA) models are multimodal robotic task controllers that, given an instruction and visual inputs, produce a sequence of low-level control actions (or motor commands) enabling a robot to execute the requested task in the physical environment. These systems face the test oracle problem from multiple perspectives. On the one hand, a test oracle must be defined for each instruction prompt, which is a complex and non-generalizable approach. On the other hand, current state-of-the-art oracles typically capture symbolic representations of the world (e.g., robot and object states), enabling the correctness evaluation of a task, but fail to assess other critical aspects, such as the quality with which VLA-enabled robots perform a task. In this paper, we explore whether Metamorphic Testing (MT) can alleviate the test oracle problem in this context. To do so, we propose two metamorphic relation patterns and five metamorphic relations to assess whether changes to the test inputs impact the original trajectory of the VLA-enabled robots. An empirical study involving five VLA models, two simulated robots, and four robotic tasks shows that MT can effectively alleviate the test oracle problem by automatically detecting diverse types of failures, including, but not limited to, uncompleted tasks. More importantly, the proposed MRs are generalizable, making the proposed approach applicable across different VLA models, robots, and tasks, even in the absence of test oracles.
