Search papers, labs, and topics across Lattice.
The paper introduces a constraint-based pre-training paradigm to enable flexible initialization of models with varying sizes, addressing the limitations of conventional pre-training. This paradigm disentangles size-agnostic knowledge into reusable weight templates and size-specific adaptation into lightweight weight scalers. The authors propose WeiT, which uses Kronecker-based constraints to regularize pre-training, achieving state-of-the-art performance in initializing models with varying depths and widths across perception and embodied learning tasks.
Stop retraining from scratch: WeiT lets you initialize models of *any* size with SOTA performance, adapting pre-trained knowledge to your specific compute budget.
The pre-training and fine-tuning paradigm has become the dominant approach for model adaptation. However, conventional pre-training typically yields models at a fixed scale, whereas practical deployment often requires models of varying sizes, exposing its limitations when target model scales differ from those used during pre-training. To address this, we propose an innovative constraint-based pre-training paradigm that imposes structured constraints during pre-training to disentangle size-agnostic knowledge into reusable weight templates, while assigning size-specific adaptation to lightweight weight scalers, thereby reformulating variable-sized model initialization as a multi-task adaptation problem. Within this paradigm, we further introduce WeiT, which employs Kronecker-based constraints to regularize the pre-training process. Specifically, model parameters are represented as compositions of weight templates via concatenation and weighted aggregation, with adaptive connections governed by lightweight weight scalers whose parameters are learned from limited data. This design enables flexible and efficient construction of model weights across diverse downstream scales. Extensive experiments demonstrate the efficiency and effectiveness of WeiT, achieving state-of-the-art performance in initializing models with varying depths and widths across a broad range of perception and embodied learning tasks, including Image Classification, Image Generation, and Embodied Control. Moreover, its effectiveness generalizes to both Transformer-based and Convolution-based architectures, consistently enabling faster convergence and improved performance even under full training.
