Search papers, labs, and topics across Lattice.
This paper argues for the development of foundation models in reinforcement learning (RL), highlighting the potential of synthetic Markov Decision Processes (MDPs) to fill the current gap in structured domains. By leveraging fixed-size sufficient statistics of MDPs, the authors propose a framework that utilizes attention-based architectures akin to those used in tabular foundation models, demonstrating that a model trained solely on synthetic MDPs can outperform traditional RL methods on benchmark tasks without task-specific tuning. The key finding reveals that this synthetic approach enables efficient learning both online and offline, significantly reducing the number of episodes required compared to existing RL techniques.
Reinforcement learning can leap forward by adopting synthetic MDPs, achieving competitive performance without the need for task-specific tuning.
Foundation models for language and vision are powered by internet-scale data, while structured domains (tabular prediction, time-series forecasting, graph learning, reinforcement learning) are not. The substitute is synthetic data, which shifts the burden from collection to prior design. Such priors already exist for many structured tasks: TabPFN and its successors solve tabular classification with a transformer pretrained on a synthetic Bayesian prior. We make two points. \textbf{First}, reinforcement learning is the conspicuous gap: sampling a synthetic MDP is as feasible as sampling a synthetic tabular dataset, yet no in-context RL work treats prior design as a primary objective. \textbf{Second}, MDPs admit a fixed-size sufficient statistic, independent of the episodes observed and tabular in shape, which makes them directly amenable to the attention-based architectures used for tabular foundation models, with a policy head replacing the supervised target. Together these define the agenda for an RL foundation model. As a proof of concept, we train one model entirely on synthetic MDPs and show that, with no task-specific tuning, it solves held-out tabular benchmarks in context, both online and offline: online, in far fewer episodes than UCB-VI and tabular Q-learning, and offline, competitively with VI-LCB.
