Search papers, labs, and topics across Lattice.
This paper investigates the limitations of Joint-Embedding Predictive Architectures (JEPAs) in text encoding, highlighting a fundamental mismatch between the squared-error latent prediction used in images and the conditional structure of language. The authors introduce three conditions鈥攑redictability, non-collapse, and low conditional variance鈥攖hat elucidate how these failures lead to centroid degeneracy and collapse pressure in text representations. Through matched experiments with I-JEPA and T-JEPA, they demonstrate that the instability and poor transfer performance in language tasks stem from the inability to accommodate multiple plausible completions, rather than a rejection of predictive learning itself.
Text encoders face a unique challenge where attempting to compress multiple valid completions into a single latent point leads to catastrophic performance issues.
Joint-Embedding Predictive Architectures (JEPAs) are effective for images, video, and audio, yet deterministic JEPA-style latent prediction has not become a standard objective for text encoders. We argue that this gap reflects a mismatch between squared-error latent prediction and the conditional structure of language. The key requirement is conditional concentration: given a context and target location, the target representation should lie near a single meaningful point. Local image prediction often satisfies this through spatial continuity, whereas masked text can admit multiple valid token or span completions whose representations need not share a coherent center. We formalize this mismatch through three conditions---predictability, non-collapse, and low conditional variance---and show how their failure creates centroid degeneracy and collapse pressure in text. Matched I-JEPA and T-JEPA experiments reveal the predicted sequence: mutual-information saturation and elevated target variance precede train--validation instability, effective-rank degeneration, cosine collapse, and poor downstream transfer. The same pattern appears across five independent data seeds, indicating that it is not a sampling artifact. These results do not rule out predictive learning for language; they show that text-compatible JEPA objectives must preserve multiple plausible completions rather than compress them into a single latent point.
