Takashi Furuya

Context-length-independent universality is mathematically possible: a single causal transformer with fixed parameters can approximate functions across arbitrary token resolutions and context lengths, provided the input sequences satisfy mild Hölder regularity.

Transformers modeling distribution-to-distribution maps hit a hard expressivity barrier on discrete atomic inputs, but achieve universal approximation in Wasserstein distance the moment input distributions satisfy a continuous level-set condition.

Lipschitz-constrained Transformers, built from gradient flows, can provably approximate any Lipschitz-continuous function, offering a path to more robust and stable architectures.