Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates whether measure-dependent pushforward models—the foundational mathematical abstraction underlying measure-theoretic transformers—can universally approximate continuous operators between spaces of probability measures. The authors show that universal approximation fundamentally fails on atomic measures because deterministic pushforwards cannot split point masses, but this obstruction vanishes under a newly formulated uniform level set condition satisfied by compact families of absolutely continuous distributions. Under this condition, they prove that measure-dependent pushforwards uniformly approximate any continuous measure-to-measure operator in the $p$-Wasserstein metric, directly establishing universality guarantees for transformer and cross-attention architectures over distributions.
Transformers modeling distribution-to-distribution maps hit a hard expressivity barrier on discrete atomic inputs, but achieve universal approximation in Wasserstein distance the moment input distributions satisfy a continuous level-set condition.
Many learning tasks map an input distribution to an output distribution. A natural way to model such an operator is to transform each input sample using a continuous function that may depend on the entire input distribution, and then take the distribution of the transformed samples. This defines a measure-dependent pushforward model and includes measure-theoretic formulations of transformers. We ask when such models can approximate arbitrary continuous operators between spaces of probability measures. We first show that universal approximation fails when atomic inputs are allowed: some continuous measure-to-measure operators that split or redistribute atomic mass cannot be approximated arbitrarily well by deterministic pushforward models. We then introduce the uniform level set condition, which requires a continuous measure-dependent scalarization whose shrinking level set neighborhoods carry uniformly vanishing mass over the input family. This condition is satisfied, in particular, by compact families of absolutely continuous measures. On every compact family satisfying this condition, we prove that any continuous measure-to-measure operator with outputs of finite $p$-th moment can be uniformly approximated, in the $p$-Wasserstein distance, by continuous measure-dependent pushforwards. Combining our theorem with existing approximation results for measure-dependent in-context maps yields universal approximation by measure-theoretic transformers. We also extend the framework to continuously-varying source measures, yielding a corresponding universality result for a class of pushforward models that are closely aligned with cross-attention architectures.