Yuval Ran-milo

Stabilization of complex linear dynamical systems becomes feasible with a memory-efficient algorithm that adapts to the system's intrinsic complexity, outperforming traditional methods.

Attention sinks aren't just a quirk of GPT-2; they're a hydra, capable of emerging through multiple distinct circuits, even when you remove the usual suspects.

Softmax attention's normalization creates unavoidable "attention sinks" when implementing trigger-conditional logic, but ReLU attention offers a sink-free alternative.