Search papers, labs, and topics across Lattice.
The paper introduces a claim-level framework using Shapley values to detect and quantify temporal knowledge leakage in LLM backtesting, addressing the issue of models inadvertently using future information. They propose Shapley-weighted Decision-Critical Leakage Rate (Shapley-DCLR) to measure the fraction of decision-driving reasoning derived from leaked information. They also introduce Time-Supervised Prediction with Extracted Claims (TimeSPEC) to filter temporal contamination by interleaving generation with claim verification and regeneration, showing it reduces leakage while maintaining performance across Supreme Court case prediction, NBA salary estimation, and stock return ranking tasks.
LLMs leak future knowledge into past predictions, but a new method called TimeSPEC can filter this temporal contamination by verifying claims and regenerating predictions based on available information.
To evaluate whether LLMs can accurately predict future events, we need the ability to \textit{backtest} them on events that have already resolved. This requires models to reason only with information available at a specified past date. Yet LLMs may inadvertently leak post-cutoff knowledge encoded during training, undermining the validity of retrospective evaluation. We introduce a claim-level framework for detecting and quantifying this \emph{temporal knowledge leakage}. Our approach decomposes model rationales into atomic claims and categorizes them by temporal verifiability, then applies \textit{Shapley values} to measure each claim's contribution to the prediction. This yields the \textbf{Shapley}-weighted \textbf{D}ecision-\textbf{C}ritical \textbf{L}eakage \textbf{R}ate (\textbf{Shapley-DCLR}), an interpretable metric that captures what fraction of decision-driving reasoning derives from leaked information. Building on this framework, we propose \textbf{Time}-\textbf{S}upervised \textbf{P}rediction with \textbf{E}xtracted \textbf{C}laims (\textbf{TimeSPEC}), which interleaves generation with claim verification and regeneration to proactively filter temporal contamination -- producing predictions where every supporting claim can be traced to sources available before the cutoff date. Experiments on 350 instances spanning U.S. Supreme Court case prediction, NBA salary estimation, and stock return ranking reveal substantial leakage in standard prompting baselines. TimeSPEC reduces Shapley-DCLR while preserving task performance, demonstrating that explicit, interpretable claim-level verification outperforms prompt-based temporal constraints for reliable backtesting.
