Gaspard Abel

Far from merely clustering tokens, full Transformers act as pattern-forming dynamical systems that natively amplify traveling and rotating waves—an inductive bias that can be engineered at initialization to accelerate training.

Ditching ReLU can provably prevent oversmoothing in deep GNNs by inducing bifurcations that create stable, heterogeneous node representations.