Search papers, labs, and topics across Lattice.
To resolve severe overconfidence issues in orthogonal parameter-efficient fine-tuning (PEFT), the authors formulate a Bayesian adaptation framework using Stein Variational Gradient Descent (SVGD) directly on the Stiefel manifold. Operating on singular value decomposition (SVD) adapter factorizations, the method transports particles along the manifold to perform posterior inference while strictly preserving matrix orthogonality constraints. The resulting geometry-aware posterior approximation significantly improves model calibration and downstream accuracy compared to conventional Euclidean SVGD and standard PEFT uncertainty estimation baselines.
Orthogonally constrained low-rank adapters produce dangerously overconfident predictions, but transporting ensemble particles directly along the Stiefel manifold via Stein variational inference recovers reliable calibration without sacrificing accuracy.
Several geometry-aware approaches to low-rank adaptation have emerged for parameter-efficient fine-tuning of large pre-trained models. These methods aim to take full advantage of the geometric structure of low-rank manifolds for improving the efficiency in subspace utilization and reducing redundancy by enforcing orthogonality constraints during optimization. The strong empirical results of these techniques have motivated further study into whether predictions from such geometry-based adaptation methods could be overconfident. In this paper, we build on the singular value decomposition factorization of adapters to develop a framework based on Stein variational gradient descent (SVGD). In this formulation, the low-rank matrices are transported along the Stiefel manifold to match the targeted distributions while retaining their crucial geometric structure. Since this geometry-aware SVGD approach provides multiple solutions during inference, it supports uncertainty quantification and produces better-calibrated adapters on the Stiefel manifold. Extensive experiments show that our method delivers strong model calibration and attains higher prediction accuracy than SVGD and related uncertainty estimation methods that are formulated in Euclidean space.
