Search papers, labs, and topics across Lattice.
This paper provides a formal proof that transformer architectures can achieve exact Bayesian posterior inference when their internal update mechanisms adhere to a Bayes joint-distribution condition. Utilizing the measure-theoretic kernel framework, the authors establish a hierarchy of abstractions, demonstrating that at each level鈥攆rom core Bayesian transformers to multilayer stacks鈥攖he update kernel aligns with the posterior almost everywhere. Key results include deriving the explicit Bayes formula for block-level architectures and validating the softmax attention mechanism as a legitimate probability distribution over keys, thereby linking abstract theory with practical implementation.
When transformers meet Bayes joint-distribution conditions, they perform rigorous Bayesian posterior updates, bridging theory and practice in AI architectures.
We present a complete formal proof that transformer architectures, when their internal update mechanisms satisfy a Bayes joint-distribution condition, implement exact Bayesian posterior inference. Working within the measure-theoretic kernel framework, we define a hierarchy of abstractions -- from the core Bayesian transformer, through semantic transformers with explicit update kernels, to full transformer blocks with QKV/attention/residual/MLP pipelines, and finally multilayer stacks -- and prove at each level that the Bayes joint semantics implies the update kernel equals the posterior almost everywhere. For the block-level architecture, we derive the explicit Bayes formula through Radon-Nikodym differentiation and prove its normalization. We additionally prove that the softmax attention mechanism induces a valid probability distribution over keys, establishing the bridge between the abstract kernel framework and concrete attention implementations. The framework makes no architectural assumptions beyond the Markov kernel structure and exposes explicit conditions under which a transformer block is provably Bayesian. In essence, when this joint distribution condition is satisfied, the forward computation of a Transformer is formally equivalent to a rigorous Bayesian posterior update.
