Search papers, labs, and topics across Lattice.
This study empirically investigates how different neural architectures respond to temporal distribution shifts across various tasks, including image classification and text regression. By employing a unified evaluation framework based on temporal drift matrices, the authors reveal that models with inductive biases favoring localized features achieve high initial accuracy but suffer from rapid degradation over time. In contrast, pretrained Transformer-based encoders, which leverage broader, more stable representations, demonstrate more gradual performance decline, highlighting the critical role of architectural choices in temporal robustness.
Models that excel in static conditions may falter dramatically as data evolves, revealing a trade-off between initial accuracy and long-term reliability.
Real-world data distributions evolve over time, inducing temporal distribution shift that can substantially degrade the reliability of deployed machine learning systems. However, the extent to which architectural choices and their associated inductive biases affect temporal robustness remains insufficiently understood. We present a systematic empirical comparison of temporal robustness across three heterogeneous, time-indexed domains encompassing image classification, multi-label text classification, and text regression tasks. Using a unified evaluation framework based on temporal drift matrices, we train models on cumulative historical data and evaluate their performance on both earlier and later time periods, thereby quantifying cross-temporal generalization. Our study spans model families ranging from simple multilayer perceptrons and convolutional networks to recurrent networks and pretrained Transformer-based encoders. Collectively, the results show that architectural inductive biases systematically shape temporal robustness: models whose inductive biases lead them to exploit localized, highly discriminative features attain the highest in-distribution accuracy, yet those features are often the ones that change most over time, so these models degrade fastest, while pretrained encoders that draw on coarser, more stable representations drift more gradually. These observations offer practical guidance for selecting architectures for real-world systems subject to temporal drift.
