Search papers, labs, and topics across Lattice.
The paper identifies a "predictive-causal gap" in representation learning where encoders trained to predict linear-Gaussian dynamics preferentially track the environment rather than the underlying system, even when this leads to causal blindness. They prove this is a structural property of the predictive objective when environment modes are slower or less noisy than system modes, and find empirical evidence of this gap in both linear and nonlinear (Duffing-GRU) systems. Operational grounding, which restricts the loss to system observables, only partially mitigates this issue.
Predictive representation learning fundamentally fails to learn causal system dynamics, instead latching onto environmental correlations, even when it hurts prediction accuracy.
We report a systematic failure mode in predictive representation learning. Across 2695 neural network configurations trained to predict linear-Gaussian dynamics, the optimal encoder tracks the environment rather than the system it is meant to model. The mean causal fidelity -- the fraction of encoder sensitivity allocated to system degrees of freedom -- is 0.49, and only 2.5% of configurations exceed 0.70. The failure intensifies with dimension: at N=100, the optimal encoder becomes causally blind (fidelity ~10^{-8}) while achieving 92% lower prediction error than the causal representation. We prove this is not an optimization artifact but a structural property of the predictive objective: when environment modes are slower or less noisy than system modes, every minimizer of the population risk encodes the former. The set of dynamics exhibiting this predictive-causal gap is open and of positive measure in parameter space. In a nonlinear Duffing-GRU sweep, unconstrained predictors learn environment-dominant representations in 55% of tasks (95% CI 41--68%) versus 24% under operational grounding (p=2.3e-3); the median out-of-distribution MSE inflation under environment shift is 1.82x versus 1.00x. Operational grounding -- restricting the loss to system observables -- partially suppresses the gap, but causal fidelity is never recovered without an explicit system-environment boundary. The results identify the predictive-causal gap as a structural limit of learning, with implications for self-supervised representation learning, world models, and the scaling paradigm.
