Timur Mudarisov

Most directions in a transformer's residual stream are functionally invisible to the terminal loss, exposing a thin observable subspace whose pullback Fisher geometry consistently outperforms standard attention weights for compression and distillation.

Key contribution not extracted.

Context expansion inflates an LLM's active token budget not to retrieve new facts, but simply to counteract softmax mass dilution caused by distractor token competition.

Geometry-guided width allocation in Transformers can lead to substantial improvements in model performance, reducing validation loss more effectively than traditional methods.

Retaining only the tangential component of the feed-forward network can preserve model quality while enhancing prediction accuracy in Transformer dynamics.