Search papers, labs, and topics across Lattice.
This paper explores the application of the Maximal Update Parametrization (渭P) to Gated Delta Networks, focusing on efficient hyperparameter tuning for large-scale language models. By analyzing the forward pass and state dynamics, the authors derive scaling rules that facilitate stable learning-rate transfer across varying model widths. Experimental results demonstrate that their approach outperforms standard parametrization methods, confirming its effectiveness in enhancing model training efficiency.
Gated Delta Networks can achieve stable learning-rate transfer across model widths, a breakthrough that significantly enhances training efficiency for large language models.
Training and scaling Large Language Models demand enormous computational resources, motivating both efficient sub-quadratic architectures and principled hyperparameter tuning methods. While the Maximal Update Parametrization (渭P) has enabled zero-shot hyperparameter transfer for standard Transformers, its extension to linear models, particularly those with structured state transitions and complicated architectures, remains largely unexplored. By rigorously propagating coordinate-size estimates through the forward pass, gating mechanisms, and recurrent state dynamics, we derive the scaling rules for Gated Delta Network. Experiments on language-model pre-training confirm that our configurations enable stable learning-rate transfer across model widths under both AdamW and SGD, whereas standard parametrization fails to transfer, validating the correctness and practical utility of our analysis.
