Search papers, labs, and topics across Lattice.
This paper introduces a framework that leverages inferred metamorphic relations (MRs) to assess the behavioral validation of UI component test suites, moving beyond traditional execution-based metrics like statement and branch coverage. By inferring component-specific MRs and aligning them with existing tests through a combination of deterministic and semantic analysis, the authors demonstrate that current test suites validate only 42.5% to 47.6% of the behavioral relations they exercise. The findings reveal significant gaps in behavioral validation, particularly in weak-oracle cases, highlighting the need for a more nuanced approach to UI testing that incorporates MR coverage metrics.
Existing UI test suites validate less than half of the behavioral relations they exercise, exposing critical gaps in testing effectiveness.
UI component libraries are commonly assessed using execution-based metrics such as statement and branch coverage, yet these metrics provide limited insight into whether tests verify the behavioral relations implied by component APIs and documentation. This paper presents an MR-based framework that uses inferred metamorphic relations (MRs) as an empirical behavioral reference, rather than a complete specification, for assessing UI component test suites. Given a component's source, documentation, and tests, the framework infers component-specific MRs using a UI-specific taxonomy, aligns tests with the inferred relations through hybrid deterministic and semantic analysis, and computes relation-level MR coverage metrics. We manually validate both the inferred MR space and the test--MR alignment. Our evaluation shows that existing test suites exercise substantially more behavioral relations than they explicitly validate: MR Cover remains between 42.5% and 47.6% across three LLM configurations and consistently below MR Touch. Most uncovered relations are weak-oracle cases, where behaviors are exercised but lack explicit behavioral validation. MR coverage also complements execution-based coverage by revealing behavioral gaps not reflected by statement or branch coverage alone. We further assess practical relevance through issue-description mapping, oracle strengthening, and MR-relevant injected faults. Most reported issue descriptions can be mapped to inferred MR relation types; weak-oracle relations often expose missing validation evidence; and MR labels show a trend in MR-relevant fault detection. Overall, MR coverage provides a complementary relation-level perspective for assessing behavioral validation in modern UI component testing.
