Search papers, labs, and topics across Lattice.
This paper addresses the hyperparameter sensitivity of Probabilistic Transformers (PTs) by applying Maximal Update Parametrization (muP) to enable efficient scaling. They demonstrate successful transfer of hyperparameters from small PT models to larger ones (up to 0.4B parameters) without retraining. The scaled PT models consistently outperform standard Transformers of comparable size on Masked Language Modeling tasks.
Probabilistic Transformers can now scale to 0.4B parameters and beat standard Transformers of the same size, thanks to a hyperparameter transfer trick.
Probabilistic Transformer (PT), a white-box probabilistic model for contextual word representation, has demonstrated substantial similarity to standard Transformers in both computational structure and downstream task performance on small models and small to medium sized datasets. However, PT is less robust to hyperparameter choices than standard Transformers, making it harder to scale efficiently. In this work, we follow Maximal Update Parametrization (muP) to rescale PT's parameters, so that hyperparameters optimized on small models can be transferred to larger models without additional tuning. With this approach, we successfully scale PT to models with up to 0.4B parameters. Experiments show that PT consistently outperforms standard transformer under the same parameter budget on Masked Language Modeling (MLM) tasks. We hope this work will contribute to the practical deployment of probabilistic models at substantially larger scales in the future.
