Mikhail Burtsev

Most directions in a transformer's residual stream are functionally invisible to the terminal loss, exposing a thin observable subspace whose pullback Fisher geometry consistently outperforms standard attention weights for compression and distillation.

Context expansion inflates an LLM's active token budget not to retrieve new facts, but simply to counteract softmax mass dilution caused by distractor token competition.

Key contribution not extracted.

Self-amplifying AI improvements can occur before visible acceleration, challenging our understanding of AI development dynamics.

MARS outperforms traditional multi-agent systems in competitive programming by harnessing specialized LLMs, achieving higher pass rates with lower costs.

Retaining only the tangential component of the feed-forward network can preserve model quality while enhancing prediction accuracy in Transformer dynamics.

Geometry-guided width allocation in Transformers can lead to substantial improvements in model performance, reducing validation loss more effectively than traditional methods.

ARMT-augmented models can handle inputs far beyond their original context limits while using 30% less compute, revolutionizing efficiency in long-context processing.