Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates how joint-embedding predictive objectives, particularly JEPA-style methods, fail to retain exogenous control-relevant features when optimizing for temporal predictability. Through a controlled experimental design, the authors demonstrate that reward-free predictive objectives consistently struggle to capture these features, achieving near chance accuracy, while a reward-grounded variant effectively recovers them with minimal labeled transitions. The findings highlight a significant limitation in current predictive learning frameworks and suggest that integrating reward signals can enhance feature retention across various environments and latent dimensions.
Reward-free predictive objectives miss crucial control-relevant features, but a mere 2% of reward-labeled data can recover them effectively.
Joint-embedding predictive (JEPA-style) objectives learn representations by predicting future latents. In doing so they can discard features that are exogenous (uncontrollable by the agent) yet control-relevant, even when those features are trivially encodable. This occurs because the objective optimizes temporal predictability rather than control-relevance. We isolate this failure mode in a controlled 2x2 experimental design that varies feature controllability and relevance independently, using a predictability knob that decouples a feature's temporal predictability from its control-relevance. Comparing six objectives: reconstruction, JEPA, action-conditioned JEPA, controllability-based JEPA, inverse dynamics under a random policy, and reward-grounded JEPA, we observe that all evaluated reward-free predictive objectives leave the exogenous control-relevant feature near chance accuracy, while a reward-grounded variant retains it selectively. The remedy is label-efficient and robust: as little as 2% of reward-labeled transitions recovers the feature, the effect holds across two environments with different surface forms, and it persists across latent dimensions from 16 to 1024. Comparing the learned latent geometry against bisimulation theory's prediction, the JEPA latent realizes only a small fraction of the class separation a supervised reference attains.