Search papers, labs, and topics across Lattice.
The paper introduces CroBo, a visual state representation learning framework that explicitly encodes "what-is-where" for robotic agents operating in dynamic environments. CroBo uses a global-to-local reconstruction objective, learning to reconstruct masked patches in a local target crop from sparse visible cues, conditioned on a global bottleneck token. This encourages the bottleneck token to capture fine-grained scene composition, leading to state-of-the-art performance on vision-based robot policy learning benchmarks and improved encoding of scene dynamics.
Encoding "what-is-where" at the pixel level in a single token unlocks SOTA performance in vision-based robot policy learning.
For robotic agents operating in dynamic environments, learning visual state representations from streaming video observations is essential for sequential decision making. Recent self-supervised learning methods have shown strong transferability across vision tasks, but they do not explicitly address what a good visual state should encode. We argue that effective visual states must capture what-is-where by jointly encoding the semantic identities of scene elements and their spatial locations, enabling reliable detection of subtle dynamics across observations. To this end, we propose CroBo, a visual state representation learning framework based on a global-to-local reconstruction objective. Given a reference observation compressed into a compact bottleneck token, CroBo learns to reconstruct heavily masked patches in a local target crop from sparse visible cues, using the global bottleneck token as context. This learning objective encourages the bottleneck token to encode a fine-grained representation of scene-wide semantic entities, including their identities, spatial locations, and configurations. As a result, the learned visual states reveal how scene elements move and interact over time, supporting sequential decision making. We evaluate CroBo on diverse vision-based robot policy learning benchmarks, where it achieves state-of-the-art performance. Reconstruction analyses and perceptual straightness experiments further show that the learned representations preserve pixel-level scene composition and encode what-moves-where across observations. Project page available at: https://seokminlee-chris.github.io/CroBo-ProjectPage.
