Search papers, labs, and topics across Lattice.
This paper investigates the impact of environmental factors (OS, Node.js version, browser) on JavaScript test flakiness by systematically executing test suites across multiple configurations. The study identified 65 environmentally flaky projects, revealing that OS, Node.js version, and browser compatibility issues are significant contributors to flakiness in JavaScript tests. To mitigate this, the authors developed js-env-sanitizer, a tool that skips and reports environmental-related flaky tests, improving CI build stability.
JavaScript tests are surprisingly brittle: changes in OS, Node.js, or browser versions can induce flakiness in a significant number of projects.
Test flakiness is a significant issue in industry, affecting test efficiency and product quality. While extensive research has examined the impact of flaky tests, many root causes remain unexplored, particularly in the context of dynamic languages such as JavaScript. In this paper, we conduct a systematic evaluation of the impact of environmental factors on test flakiness in JavaScript. We first executed test suites across multiple environmental configurations to determine whether changes in the environment could lead to flaky behavior. We selected three environmental factors to manipulate: the operating system, the Node.js version, and the browser. We identified a total of 65 environmental flaky projects, with 28 related to operating system issues, five to Node.js version compatibility, 16 to a combination of operating system and Node.js issues, and 17 related to browser compatibility. To address environmental flakiness, we developed a lightweight mitigation approach, js-env-sanitizer, that can sanitize environmental-related flaky tests by skipping and reporting them (rather than failing), allowing CI builds to continue/succeed without rerunning entire test suites. The tool achieves high accuracy with minimal performance or configuration overhead, and currently supports three popular JavaScript testing frameworks (Jest, Mocha, and Vitest)
