Search papers, labs, and topics across Lattice.
This paper introduces Surprise-based Nested Temporal Abstraction (SUNTA), a novel approach to hierarchical video prediction that utilizes prediction errors to define chunk boundaries, addressing limitations of traditional fixed-length and similarity-based methods. By employing a decoupled training strategy and leveraging internal inconsistency as a surprise metric, SUNTA effectively maintains the integrity of surprise signals during training and prediction. Experimental results show that SUNTA significantly outperforms existing baselines, achieving accurate predictions over 250 timesteps, while other methods fail within the first 10 timesteps.
SUNTA achieves unprecedented long-horizon video predictions by redefining chunk boundaries based on prediction errors, outperforming traditional methods that falter early.
Hierarchical state-space models (HSSMs) offer a promising approach to long-horizon prediction by segmenting sequences into temporal chunks. However, their performance hinges on how chunk boundaries are determined. While prior HSSMs typically rely on fixed-length chunking or similarity-based boundary detection, these methods often misalign with the intrinsic temporal structure of the data. We argue that chunking should instead be driven by prediction errors, which more directly indicate when longer-range context becomes necessary. Nevertheless, integrating surprise-based chunking into HSSMs introduces critical challenges, including hierarchical collapse during end-to-end training and the absence of surprise signals during open-loop prediction. To address these issues, we propose Surprise-based Nested Temporal Abstraction (SUNTA), a method that employs a decoupled training strategy to preserve surprise signals and uses internal inconsistency as a top-down surprise metric to determine chunk boundaries within imagined rollouts. Experiments on video prediction tasks in 2D and 3D environments demonstrate that SUNTA outperforms baselines, uniquely maintaining accurate predictions over 250 timesteps, whereas all baselines degrade within the first 10 timesteps.
