Search papers, labs, and topics across Lattice.
This paper introduces Observed Transition Factorization (OTF), a method that decomposes complex observation transitions in multi-object scenes into a sparse set of reusable transition primitives, enabling the robust formation of action-like latents. By leveraging these primitives, the authors develop OTF-LAM and OTF-LAM-Dino, which effectively abstract motion into action-like representations within an inverse-forward dynamics framework. Empirical results demonstrate that OTF primitives exhibit zero-shot transferability across various scene conditions and significantly enhance policy learning performance in ambiguous transition scenarios.
Transition effects can be disentangled into reusable primitives, leading to superior policy learning even in complex, ambiguous environments.
Latent Action Models (LAMs) learn action-like proxies from observation transitions. However, in multi-object or distractor-rich scenes, these visual effects mix agent motion with distractors, camera dynamics, and background changes, making the underlying action source ambiguous without supervision. Structuring this mixture as reusable transition effects provides an intermediate representation from which action-like latents can be more robustly formed. We introduce Observed Transition Factorization (OTF), which decomposes each transition into a sparse set of observed transition primitives. Using these primitives as the transition interface, we propose OTF-LAM, which abstracts motion primitives into action-like latents within the standard inverse-forward dynamics framework, and OTF-LAM-Dino, a decoder-free variant that predicts future states in a frozen DINOv2 representation space. Empirically, OTF primitives transfer zeroshot across controlled carrier and morphology shifts, showing reusability. Furthermore, downstream policy learning results match or outperform baselines under complex transition ambiguity.
