Search papers, labs, and topics across Lattice.
This paper investigates the under-explored problem of test flakiness in REST API fuzzing, using EvoMaster as a representative fuzzer and a corpus of 36 REST APIs. Through empirical analysis of nearly 3000 failing tests, the authors categorize and analyze the sources of flakiness. Based on these insights, they propose FlakyCatch, a novel approach for detecting and mitigating flakiness, and demonstrate its effectiveness with both white-box and black-box fuzzers.
REST API fuzzing, a critical component of modern software development, suffers from significant flakiness issues that can now be reliably detected and mitigated.
Test flakiness is a common problem in industry, which hinders the reliability of automated build and testing workflows. Most existing research on test flakiness has primarily focused on unit and small-scale integration tests. In contrast, flakiness in system-level testing such as REST APIs are comparatively under-explored. A large body of literature has been dedicated to the topic of fuzzing REST APIs, whereas relatively little attention has been paid to detecting and possibly mitigating negative effects of flakiness in this context. To fill this major gap, in this paper, we study the flakiness of tests generated by one of the popularly applied REST API fuzzer in the literature, namely EvoMaster, conduct empirical studies with a corpus of 36 REST APIs to understand flakiness of REST APIs. Based on the results of the empirical studies, we categorize and analyze flakiness sources by inspecting near 3000 failing tests. Based on the understanding, we propose FlakyCatch to detect and mitigate flakiness in REST APIs and empirically evaluate its performance. Results show that FlakyCatch is effective in detecting and handling flakiness in tests generated by white-box and black-box fuzzers.
