Search papers, labs, and topics across Lattice.
This paper introduces variational neurons into Transformer feed-forward networks, replacing deterministic units with local variational units based on the Evidential Variational Evidence (EVE) framework. The authors evaluate this approach on next-token language modeling tasks, comparing variational and deterministic variants using predictive and probabilistic metrics. Results demonstrate that variational neurons integrate stably, maintain strong predictive performance, and generate informative uncertainty signals, establishing variational Transformers as a practical approach for uncertainty-aware language modeling.
Transformers can now predict with an explicit internal structure of uncertainty, enabling stronger probabilistic evaluation and a more informative analysis of model behavior.
Transformers for language modeling usually rely on deterministic internal computation, with uncertainty expressed mainly at the output layer. We introduce variational neurons into Transformer feed-forward computation so that uncertainty becomes part of the internal computation itself. Concretely, we replace deterministic feed-forward units with local variational units based on EVE while preserving the overall Transformer backbone. We evaluate this design in compact next-token language-modeling settings. We compare deterministic and variational variants with both predictive and probabilistic criteria. Alongside negative log-likelihood, perplexity and accuracy, we analyze calibration, conditional variance, mutual information and latent-usage statistics. The resulting picture is clear. Variational neurons integrate stably into Transformers, preserve strong predictive performance and produce informative uncertainty signals. The experiments also show that task quality, useful depth and internal stability are distinct properties. These results establish variational Transformers as a practical form of uncertainty-aware language modeling. They show that Transformers can predict with an explicit internal structure of uncertainty, which supports stronger probabilistic evaluation and a more informative analysis of model behavior.
