Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a normalized signed primal integral for evaluating anytime solvers, addressing critical decisions regarding scoring runs without feasible solutions and the treatment of reference values. By comparing different scoring methods, the authors reveal that their proposed approach maintains panel orderings while avoiding pitfalls associated with common acceptance thresholds. The findings demonstrate that updating reference values can significantly alter score levels without affecting the overall ranking of solutions, highlighting the importance of careful reference policy choices in solver evaluation.
A new evaluation method reveals that updating reference values can drastically change score levels while preserving solution rankings, challenging conventional scoring practices.
Anytime solvers return usable solutions before they terminate and improve them while time remains. Their progress is commonly summarized by a primal integral, a final gap, or convergence curves. Each summary leaves consequential choices open: how to score a run before its first feasible solution, whether poor incumbents are truncated, and whether the reference value is updated after the experiment or version-frozen beforehand. These choices become visible when runs produce no valid incumbent or improve a published best-known value. We study a normalized signed primal integral built from a bounded relative gap. It assigns an intrinsic worst value to an empty run, requires no acceptance threshold, and assigns negative instantaneous gaps to incumbents that beat a frozen reference. We compare the smooth difference-over-sum kernel with a signed version of Berthold's max-normalized gap and use the former as a working default. We also distinguish analysis-time from version-frozen reference policies and recommend reading the score with the raw final gap, mean convergence curve, and target-attainment curve. We evaluate these choices on a five-arm routing campaign and two model-fidelity ladders. The two signed kernels preserve every panel ordering in this study, whereas a common acceptance threshold compresses the distances between arms and reverses one panel ordering. Updating 54 of the 212 reference values changes score levels and removes all negative scores, while leaving the observed panel orderings unchanged. An exploratory screen also identifies nine panel comparisons in which similar integral scores conceal materially different endpoints or attainment rates. The implementation, frozen inputs, and generators are openly released.