Search papers, labs, and topics across Lattice.
This paper introduces Successive Capacity Growth (SCG), a method for dynamically expanding Vision Transformer encoders in Joint-Embedding Predictive Architectures (JEPAs) based on task complexity. By starting with a minimal architecture and incrementally adding width or depth through a test-and-verify mechanism, SCG achieves a 20.3% reduction in prediction loss on a multi-object dynamics task and a 23% improvement on a 2D navigation task compared to fixed-size models, all while maintaining high parameter efficiency. The approach ensures that learned semantic dimensions remain independent and aligned with predictive objectives, allowing for efficient scaling without compromising performance.
SCG enables Vision Transformer encoders to adaptively grow in complexity based on task demands, achieving significant efficiency gains without sacrificing representation quality.
Joint-Embedding Predictive Architectures (JEPAs) for world modeling typically employ fixed-size Vision Transformer encoders that are over-provisioned for simple tasks and under-provisioned for complex ones, with significant redundancy across attention heads. We propose Successive Capacity Growth (SCG), a method that starts from a minimal encoder (1 head, 2 layers, 283K parameters) and grows incrementally in width (adding attention heads for low-level semantic capacity) or depth (adding transformer blocks for higher-order semantic abstraction), driven by a task-agnostic test-and-verify mechanism that exploits function-preserving expansion to safely trial architectural changes and roll back if they do not improve prediction loss. The Sketched Isotropic Gaussian Regularizer (SIGReg) ensures that all learned semantic dimensions remain statistically independent and aligned with the predictive objective, preventing collapse even as the architecture grows. On a 60-dimensional multi-object dynamics task, SCG naturally triggers depth expansion, improving prediction loss by 20.3% over the fixed small baseline with 56 times greater parameter efficiency than scaling to the fixed large model; on a 2D navigation task, a single width expansion yields even an 23% improvement over the fixed large model. Across all three tested environments of increasing complexity, the adaptive encoder matches or exceeds the fixed small baseline, with zero false-positive expansions and bit-exact function preservation (ratio = 1.0, absolute difference = 0.0). The take-away is that JEPA world model encoders need not be pre-allocated at maximum capacity - they can grow successively as the task demands, achieving significant compute and data efficiency while maintaining representation quality.
