Search papers, labs, and topics across Lattice.
This study investigates the impact of faults in OpenAPI specifications on the effectiveness of black-box testing tools in microservice systems. By introducing a taxonomy of six fault classes and injecting faults at five severity levels, the researchers evaluated the performance of three testing tools across two benchmarks, revealing significant and varied degradation patterns in testing effectiveness. Key findings indicate that while some faults, particularly in method semantics, lead to widespread degradation across metrics, others, like schema constraint relaxations, can cause hidden issues that affect request/response quality without altering coverage metrics.
Faults in OpenAPI specifications can drastically undermine black-box testing effectiveness, revealing hidden vulnerabilities that traditional coverage metrics fail to capture.
OpenAPI specifications are the primary input for black-box testing tools in microservice systems (MSS), yet prior work shows these specifications are often incomplete, inconsistent, or incorrect. Despite this, most studies on OpenAPI-based black-box testing assume correct specifications and evaluate tool performance. We address this gap by introducing a literature-grounded taxonomy of six OpenAPI specification fault classes. We inject faults at five severity levels, and evaluate the resulting mutated specifications on two microservice benchmarks, TrainTicket and SocialNetwork, using three testing tools: EvoMaster, RESTler, and Schemathesis. We measure the impact of these faults using code coverage, specification coverage, request/response quality, and behavioral diversity. Our results show that specification faults cause strong and heterogeneous degradation patterns across testing tools and systems. Faults in method semantics cause broad degradation across all metrics, while others, such as modifications to response codes, remain weak. Relaxations of schema constraints cause hidden degradation, with no impact on code and specification coverage but a large impact on request/response quality. These findings demonstrate that specification quality directly shapes black-box API testing effectiveness. Also, code and specification coverage-only evaluations can understate the impact of specification faults on black-box testing in MSS and should be complemented by request/response quality and behavioral diversity.
