Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Kepler-Encoder-v0.1, a multimodal encoder designed for robots that integrates visual, proprioceptive, and force/torque data into a unified latent space using a learned-query cross-attention layer. The model is trained self-supervised through masked cross-modal prediction, demonstrating its ability to recover end-effector states and force information significantly better than traditional vision-only approaches, particularly in scenarios where visual input is limited. Notably, the encoder's frozen latent representation serves as an effective invalid-state monitor and can reconstruct camera frames, highlighting its practical utility across multiple robotic embodiments.
Fusing robot state into a multimodal encoder allows for superior recovery of force and motor states, even when relying solely on visual input.
A robot must understand the state of its own body, but a camera sees only part of it. Force and contact leave almost no trace in a single frame, and raw vision features read force at $R^2$ at or below $0.10$ on every robot we test. We present Kepler-Encoder-v0.1, a robot-first multimodal encoder that treats robot state as a modality and fuses vision, proprioception, and force/torque into a single shared latent with a learned-query cross-attention layer, trained self-supervised by masked cross-modal prediction under the LeJEPA/SIGReg objective. At evaluation only vision enters, which poses a sharp question. Does fusing state into training make the vision-only latent carry anything the pixels do not already contain? On the RH20T corpus the answer is yes, precisely where the camera is weakest. On held-out scenes, the vision-only latent recovers end-effector state, and force in particular, significantly above both raw frozen-ViT features and a compute-matched vision-only control on every sensored robot, though absolute force recovery at a single timestep is modest; on motor state, which the camera largely sees, it is statistically tied with the strongest vision baselines, and it is the only feature whose latent geometry tracks state. A single embodiment-agnostic encoder covers four robots, and a data-matched control shows this breadth reflects embodiment diversity rather than data volume. The frozen latent is directly useful. Its own cross-modal prediction error is a training-free invalid-state monitor (AUROC $0.90$ on out-of-range states, $0.69$ on scene-swapped states), and a diffusion decoder (PixNerd) reconstructs the camera frame from the latent, confirming the spatial compression preserves world-state. This report validates the single-timestep case; native-rate temporal fusion is the next step.