Radu State

Most directions in a transformer's residual stream are functionally invisible to the terminal loss, exposing a thin observable subspace whose pullback Fisher geometry consistently outperforms standard attention weights for compression and distillation.

Key contribution not extracted.

Context expansion inflates an LLM's active token budget not to retrieve new facts, but simply to counteract softmax mass dilution caused by distractor token competition.