Chenxiao Yang

A simple two-layer transformer with $O(\log\log N)$-bit precision can perfectly copy arbitrary strings up to length $N$ and terminate correctly, a feat impossible for standard transformers with finite-precision softmax.

Transformers require a surprisingly high number of examples for effective chain-of-thought learning, challenging assumptions about their efficiency.

Recursive models can exponentially shrink the context window needed for long-horizon reasoning, blowing past the limitations of even the biggest LLMs.