Search papers, labs, and topics across Lattice.
This paper investigates the effectiveness of anytime-valid inference using conformal test martingales for monitoring real-time forecasting streams, particularly in scenarios where data dependencies challenge traditional assumptions. The authors conduct a case study with a Kalman adapter to correct time-series models, revealing that while the method performs well on synthetic data, it fails consistently on real data due to the inherent characteristics of the deployed score stream. The findings highlight the necessity of incorporating null-calibration controls and mechanism traces to mitigate the unintended consequences of gating in dependent data environments.
Real-time monitoring can fail catastrophically when the underlying data is dependent, as shown by a 100% failure rate in real forecasting streams despite theoretical guarantees.
Machine learning systems are increasingly corrected while they run, and the decision of when to intervene is increasingly delegated to statistical monitors. Anytime-valid inference promises evidence that can be acted on at any moment, exactly the guarantee this setting needs, and it is moving from theory into deployed monitoring. Conformal test martingales are the change-detection instrument, and Ville's inequality caps their false-alarm probability on exchangeable data. The guarantee is conditional. A deployment inherits it only if the stream it monitors behaves exchangeably. The premise is hardest to satisfy where these monitors are most useful, on dependent data and inside loops where the monitor modifies the learner whose scores it reads. It is also rarely measured. We measure it in a pre-specified case study, where such a monitor gates the online updates of a Kalman adapter correcting frozen time-series foundation models on five forecasting streams. On exchangeable synthetic streams, the same implementation fires in at most 1 of 60 runs. On the real streams, at alpha = 0.05, 135 of 135 clean-stream runs fired. The construction does not explain the firing; the failure comes from the deployed score stream itself. Repeated fires hold the gate's drift response active, and the gated filter amplifies the very transient it was designed to prevent. The component worth keeping makes no validity claim. Huber-style gating of the filter's own updates cuts isolated-spike degradation by an order of magnitude with no dataset specific tuning. Anytime-valid methods proposed for dependent data should therefore be accompanied by null-calibration controls and mechanism traces.
