Search papers, labs, and topics across Lattice.
This paper explores the integration of large language models (LLMs) into forecasting systems, categorizing them into standalone, tool-augmented, and hybrid architectures that leverage temporal data and external evidence for predictive tasks. The study highlights the importance of rigorous training and evaluation methods, revealing that measurement inaccuracies and sensitivity to input variations significantly hinder performance. Key findings indicate that while LLMs can enhance forecasting accuracy, issues such as contamination and the need for robust calibration under distribution shifts remain critical challenges to address.
LLMs can significantly boost forecasting accuracy, but their effectiveness is hampered by measurement limitations and sensitivity to input changes.
Large language models (LLMs) now support forecasting systems that combine language-based reasoning with temporal data, evidence retrieval, external tools, and iterative prediction. We investigate LLM-based forecasting agents, meaning systems in which a language model contributes to a scored prediction about a future or currently unobserved target. We organize architectures into three groups. Standalone LLM workflows operate on encoded time series or event context. Tool- and retrieval-augmented agents incorporate external evidence. Hybrid systems pair LLMs with statistical or foundation models. We then review training methods and evaluation protocols. We examine negative as well as positive evidence, including sensitivity to small input perturbations, ablations in which the LLM component does not improve accuracy, and benchmark gains that may reflect contamination instead of temporal reasoning. We cover applications in finance, weather, health, energy, and operations, and we summarize the benchmarks and datasets used for evaluation. The evidence indicates that measurement is a central limitation. Future work requires calibration under distribution shift, contamination-resistant live evaluation, explicit reporting of cost and accuracy together, and methods for handling feedback between deployed forecasts and the outcomes being forecast.
