Itay Lavie

Decoupling update magnitude from gradient norm in SGD initially accelerates learning but leads to marginal stability and distinct scaling regimes determined by model width and batch size.

Neural scaling laws reveal unexpected insights into how deep learning models defy classical statistics, reshaping our understanding of model performance in scientific applications.

A first-order phase transition in softmax attention reveals how copy mechanisms emerge abruptly during training, challenging existing notions of gradual feature learning.