Search papers, labs, and topics across Lattice.
This paper introduces Hindcast, a novel evaluation framework for large language models (LLMs) that mitigates biases in forecasting assessments by simulating a past evaluation environment. By utilizing a frozen snapshot of public Reddit posts and replaying resolved Polymarket prediction markets, Hindcast ensures that models are graded on their foresight without the influence of post-event information. The key finding reveals that while retrieval aids model performance in contexts with prior discussions, it can detrimentally impact accuracy when only speculative content is available.
Hindcast reveals that LLMs can be misled by speculative content, showing that retrieval-based evaluation can backfire when prior discussions are absent.
Forecasters are evaluated by backtesting, which replays resolved questions and grades the probability the system would have assigned before the outcome was known. For LLMs, two channels leak the answer into this test. A model that retrieves can surface reports written after the event, turning forecasting into a lookup, and each new model is trained on data closer to the event, so a question that lay in the future for last year's models sits inside this year's training data. Either way, the test grades recall while claiming to grade foresight. We introduce Hindcast, which closes both leaks by grading a model as if it stood at a chosen past date $t_0$, before the outcome existed in either channel. Hindcast replays resolved Polymarket prediction markets against a frozen snapshot of public Reddit, lets the model read only posts written before $t_0$, and scores each forecast against both what happened and the market's own price at $t_0$, itself a human forecast made from the same past information. Because the cutoff is set per market and the snapshot never changes, the evaluation re-runs on new markets as models improve, without going stale. Once the leak is closed, retrieval still helps most models, but only where Reddit discussed the event beforehand. Where the archive carried only speculation, retrieval hurts.
