Search papers, labs, and topics across Lattice.
The paper introduces Predictive State Retrieval (PSR), a novel task where a model uses a short video prefix and a temporal question to retrieve relevant instances from other videos or images that depict an object's future state. By constructing a benchmark from four datasets with human-validated ground truth, the authors reveal that while the true future state is retrievable once specified, existing predictors, including a large multimodal language model, fall significantly short of the oracle performance. The proposed LFTR retriever effectively narrows this gap by leveraging cross-space fusion and hard-negative training, achieving lower inference costs while addressing the primary challenge of forecasting over perception.
Forecasting future states from video prefixes reveals a significant gap in current models' retrieval capabilities, with LFTR leading the way in bridging this divide.
We introduce Predictive State Retrieval (PSR), a task in which a model observes a short video prefix and a temporal question about an object's future state, then retrieves instances from other videos or images that depict that state. Unlike action anticipation, which predicts a label, moment retrieval, which localizes an observed event within a video, or video generation, which synthesizes pixels, PSR combines anticipation with cross-instance retrieval across multiple temporal horizons. We construct a benchmark from four datasets with graded, human-validated ground truth, difficulty tiers, and an oracle ceiling. We also propose LFTR, a lightweight retriever with frozen encoders that predicts a question- and horizon-conditioned future latent and matches it in complementary semantic and visual spaces. A ceiling decomposition reveals a clear bottleneck: the true future state is highly retrievable once specified, whereas every predictor we evaluate, including a large multimodal language model with access to the prefix frames, remains far below the oracle. Thus, forecasting rather than perception is the central learnable challenge. LFTR narrows this gap at substantially lower inference cost, and ablations attribute its gains to cross-space fusion and hard-negative training rather than latent rollout. We release the benchmark, code, and evaluation scripts.
