Search papers, labs, and topics across Lattice.
To determine whether time-series foundation model performance survives strict temporal separation, the authors evaluated 13 forecasters across seven dataset groups from five domains where all test observations postdate the release of every model. Pretrained models won five of seven groups, but their competitive edge was completely uncorrelated with intrinsic series properties like spectral entropy or seasonality, tracking instead whether the domain appeared in their pretraining data (e.g., TimesFM securing its largest margin on Wikipedia pageviews, its primary pretraining corpus). The results prove that temporal hold-outs prevent memorization of specific windows but fail to isolate out-of-domain generalization, establishing that model utility is governed by pretraining corpus familiarity.
Post-cutoff test sets only prevent verbatim window memorization, revealing that time-series foundation model outperformance is driven by pretraining corpus familiarity rather than generalized temporal reasoning.
Time-series foundation models are evaluated almost exclusively on public archives that predate them, so a strong score cannot be separated from having seen the test set during pretraining. The obvious remedy is a hold-out that postdates the models. We build one: thirteen forecasters -- four classical, three trained per dataset, six pretrained -- on seven groups drawn from five domains, every observation published after the last model was released, and every dataset rebuildable without an API key. Under this protocol pretrained models win 5 of 7 groups, lose one to a Theta baseline, and on daily exchange rates are indistinguishable from a seasonal naive forecast, along with every other method tested. We then ask what separates the wins from the losses, and report a negative result: the two intrinsic properties one would reach for -- seasonal strength and spectral entropy, measured on the input window -- do not account for the pattern, and seasonal strength is if anything negatively associated with the advantage. What does track it is corpus familiarity. Our largest gain (28% lower MASE than the best classical method, on weekly Wikipedia pageviews) falls on Wikipedia pageviews, the domain TimesFM's authors describe as the bulk of its pretraining corpus, at the same granularities and differing only in time window. Within the pretrained family, where every model forecasts identical series so that series difficulty cancels, the TimesFM family outranks the Chronos family by -0.53 ranks on Wikipedia against -0.09 everywhere else (1,500 vs. 754 series, Mann-Whitney p < 1e-5). We conclude that a temporal hold-out removes memorisation of a window but not familiarity with a domain, that benchmarks therefore need domain hold-outs stated relative to disclosed corpora, and that the practitioner's question is less which model is better than whether their domain is one the model was raised on.
