Track new results across pure and applied mathematics.
Search papers, labs, and topics across Lattice.
Addressing the limitation where standard expressivity proofs require distinct network weights for each sequence length, this work proves that a single causal transformer with fixed parameters can uniformly approximate causal maps over arbitrary context lengths on a normalized horizon. By treating token sequences through the lens of $\alpha$-Hölder continuity, the authors establish an infinite-length mean-field limit where masked self-attention converges to a continuous causal time integral. Crucially, they derive an empirical risk minimization generalization bound of $O((\log\log N/\log N)^{\beta/(d+2)})$ that depends strictly on sample size $N$ and token dimension $d$, entirely free of any maximum context length factor.
Context-length-independent universality is mathematically possible: a single causal transformer with fixed parameters can approximate functions across arbitrary token resolutions and context lengths, provided the input sequences satisfy mild Hölder regularity.
Long contexts are central to modern transformer systems, but most expressivity results choose a different network for each fixed sequence length. We study whether one masked transformer can approximate causal token-to-token maps uniformly over sequences of arbitrary length sampling a fixed normalized horizon. To relate sampling resolutions, we model tokens by $α$-Hölder sequences or, more generally, a common modulus of continuity. Our notion of continuity across resolutions characterizes the causal families admitting uniform approximation on these compact input classes by a single transformer with length-independent parameters. The result extends to the infinite-length mean-field limit, where tokens form continuous curves and masked attention becomes a causal time integral. For bounded regression with target maps satisfying a $β$-smooth stability condition defined using regular test functions, quantitative approximation yields a generalization bound: exact empirical risk minimization over suitably sized bounded-weight transformers gives root mean-square prediction error $O((\log\log N/\log N)^{β/(d+2)})$ from $N$ iid labeled sequences. The bound holds at fixed confidence on the same sampling distribution, with $d$ the token dimension and no maximum-length factor. Finally, experiments on physical time series support the Hölder-regular token model at observed scales, with dataset-dependent fitted exponents, whereas text input embeddings provide a contrasting case. Native and dense sampling, shuffled controls, and refinement checks delimit this empirical regularity regime.