Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper establishes finite-sample generalization bounds for Transformers by framing their training as a finite-horizon Markovian control problem, utilizing a measure-valued approach to interpret datasets as probability laws. By quantizing the model's state, action, and measure-state spaces, the authors derive explicit bounds using concentration inequalities and a Lipschitz stability estimate for the value function, which are then transferred to the base model with a manageable approximation error. Additionally, the study connects Transformer generalization to Wasserstein distributionally robust optimization, highlighting a novel perspective on training dynamics and robustness.
Finite-sample generalization bounds reveal a surprising link between Transformer training and Wasserstein distributional robustness, reshaping our understanding of model dynamics.
We derive finite-sample generalization bounds for Transformers trained with dynamic programming recursions. Building on the doubly lifted, measure-valued formulation of Transformer dynamics, we view data sets as probability laws on pairs of empirical input-output measures, allowing us to interpret the training problem as a finite-horizon Markovian control problem. We then analyze a quantized model, derived by quantizing the state, action, and measure-state spaces, and derive explicit finite-sample generalization bounds using concentration inequalities for empirical laws on finite metric spaces together with a Lipschitz stability estimate for the value function. These bounds are transferred to the base model at the cost of an explicit approximation error. Finally, we show that the same machinery yields a distributionally robust control formulation of the training problem, connecting Transformer generalization to Wasserstein distributionally robust optimization.