Search papers, labs, and topics across Lattice.
This paper analyzes the limitations of traditional closed-world software testing assumptions when applied to open-world games, highlighting challenges like inexhaustible behavior spaces and unstable test oracles. It proposes a shift towards testing methodologies that characterize and interpret system behavior under uncertainty, rather than relying on deterministic validation. The authors outline research directions for automated test generation, evaluation metrics, and empirical study design tailored for these dynamic environments.
Closed-world testing assumptions crumble when faced with the infinite possibilities of open-world games, demanding a new paradigm focused on characterizing behavior under uncertainty.
Software testing research has traditionally relied on closed-world assumptions, such as finite state spaces, reproducible executions, and stable test oracles. However, many modern software systems operate under uncertainty, non-determinism, and evolving conditions, challenging these assumptions. This paper uses open-world games as an extreme case to examine the limitations of closed-world testing. Through a set of observations grounded in prior work, we identify recurring characteristics that complicate testing in such systems, including inexhaustible behavior spaces, non-deterministic execution outcomes, elusive behavioral boundaries, and unstable test oracles. Based on these observations, we articulate a vision of software testing beyond closed-world assumptions, in which testing supports the characterization and interpretation of system behavior under uncertainty. We further discuss research directions for automated test generation, evaluation metrics, and empirical study design. Although open-world games serve as the motivating domain, the challenges and directions discussed in this paper extend to a broader class of software systems operating in dynamic and uncertain environments.
