Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces ProtocolMatch, a novel evaluation framework for scientific dynamics forecasting that explicitly accounts for protocol-dependent model selection beyond just architecture choice, incorporating factors like observed history, rollout feedback, and compute budget. By comparing recurrent, patched-attention, causal-attention, and low-rank linear predictors on driven quantum-spin dynamics across varied datasets and conditions, the study reveals that optimal model choices are highly sensitive to specific task parameters and evaluation protocols. Key findings include the reversal of performance ordering between causal-attention and recurrence with increasing training data, the superiority of linear predictors in certain complex systems, and the critical impact of history observation and state refresh on predictive accuracy and robustness.
Model selection for scientific forecasting is fundamentally protocol-dependent, with optimal choices reversing based on dataset size, system complexity, and evaluation metrics like state refresh versus closed-loop performance.
Scientific dynamics forecasting is often framed as an architecture choice, although deployment is also determined by observed history, rollout feedback, compute budget, physical objective, and test distribution. We formulate protocol-dependent model selection and introduce ProtocolMatch, a compute-matched, validation-selected, and failure-preserving evaluation framework. On driven quantum-spin dynamics, we compare recurrent, patched-attention, causal-attention, and low-rank linear predictors across three independently generated datasets. The causal-attention--recurrence ordering reverses as the training set grows within a fixed two-spin task, while a linear predictor has the lowest mean error in the six-spin local-observable comparison. Restricting observed history worsens every refreshed-history view but improves every closed-loop view in the four-spin study. A latest-state MLP has lower error than persistence on every dataset under state refresh across all five cells, yet its closed-loop rank varies by system and includes finite explosive errors. Physical penalties improve targeted consistency without reliably improving prediction error, and in-distribution intervals lose most coverage after a driving-frequency shift. Thus scientific model selection should return a predictor with its protocol and report accuracy, physical validity, and shifted-distribution reliability separately.