Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper critiques traditional benchmarking methods in stateful environments, highlighting how adaptive mechanisms introduce temporal dependencies that bias performance estimations. The authors advocate for a shift towards evaluating performance differentials rather than absolute measures, formalizing benchmarking as a decision problem focused on identifying the fastest program. They propose experimental designs that yield consistent estimators of performance contrasts, enabling accurate benchmarking even with finite resources.
Performance differentials, not absolute metrics, are the key to accurate software benchmarking in stateful environments.
In the perpetual pursuit of performance, modern computing systems rely ever more on stateful mechanisms to accommodate the dynamics of workloads and physical environments, bolstering efficiency but confounding benchmarking and thereby the optimization of software. Indeed, by their nature, adaptive mechanisms introduce temporal dependencies between measurements and render naive estimators of individual program performance biased. Observing that rectifying such biases necessitates speculative assumptions about system dynamics, we call for prioritizing performance differentials over absolute measures and formalize software benchmarking as the decision problem of identifying the fastest program, for which relative knowledge suffices. To this end, we propose simple experiment designs admitting consistent estimators of contrasts, whereby program-specific biases cancel under tenable assumptions. These designs asymptotically yield the correct decision and afford a robust methodology for finite-budget benchmarking in stateful environments, bearing broad implications for the development of performance-sensitive software.