Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper identifies a critical flaw in database benchmarking: compiler-induced variations in baseline performance across builds, even with unchanged source code, leading to unreliable comparisons. To address this, they introduce Multi-Version Experimental Evaluation (MVEE), which automatically detects and materializes different compiled versions of methods based on assembly code analysis. By including all observed versions in the evaluation, MVEE significantly improves the quality and expressiveness of experimental results.
Compiler optimizations can subtly but significantly alter baseline performance across builds, invalidating typical database benchmarking practices.
In the database community, we typically evaluate new methods based on experimental results, which we produce by integrating the proposed method along with a set of baselines in a single benchmarking codebase and measuring the individual runtimes. If we are unhappy with the performance of our method, we gradually improve it while repeatedly comparing to the baselines, until we outperform them. While this seems like a reasonable approach, it makes one delicate assumption: We assume that across the optimization workflow, there exists only a single compiled version of each baseline to compare to. However, we learned the hard way that in practice, even though the source code remains untouched, general purpose compilers might still generate highly different compiled code across builds, caused by seemingly unrelated changes in other parts of the codebase, leading to flawed comparisons and evaluations. To tackle this problem, we propose the concept of Multi-Version Experimental Evaluation (MVEE). MVEE automatically and transparently analyzes subsequent builds on the assembly code level for occurring "build anomalies" and materializes them as new versions of the methods. As a consequence, all observed versions of the respective methods can be included in the experimental evaluation, highly increasing its quality and overall expressiveness.