Lenka Zdeborová

Kernel learners outperform feature learners for nonlinear in-context learning when task diversity is high and context length is short, but the opposite holds for low diversity and long contexts.

Attention parameterization can trap models in uninformative states, but symmetry-breaking mechanisms can dramatically enhance recovery efficiency.