Anxin Guo

A simple two-layer transformer with $O(\log\log N)$-bit precision can perfectly copy arbitrary strings up to length $N$ and terminate correctly, a feat impossible for standard transformers with finite-precision softmax.