Search papers, labs, and topics across Lattice.
5
0
7
Stochastic training can transform Looped Transformers from brittle to robust, dramatically reducing OOD variance and improving performance across diverse tasks.
Separating the magnitude and direction of weight vectors can lead to more predictable training dynamics and significant performance gains across various neural network architectures.
Tying expert parameters across layers can halve memory usage in MoE models without sacrificing performance, revolutionizing how we scale LLMs.
Multilingual data quality classifiers can outperform monolingual ones, but only with careful tuning of the decision boundary, challenging the assumption that scale alone guarantees improved filtering.
Training a 70B parameter open-source LLM on a supercomputer reveals the hidden engineering hurdles and infrastructure adaptations needed to democratize large-scale AI development beyond the private sector.
