Search papers, labs, and topics across Lattice.
This paper introduces Fisher8, a novel output-layer gradient correction method that leverages Fisher geometry to stabilize neural heteroscedastic regression by better aligning gradient updates with the local curvature of the loss landscape. By avoiding data-dependent hyperparameters and incorporating an approximate KL trust radius, Fisher8 enhances the convergence of neural networks in predicting both mean and uncertainty from noisy observations. Experimental results demonstrate that Fisher8 achieves improved likelihood-error tradeoffs and generates calibrated uncertainty estimates across various multidimensional regression and representation-learning tasks.
Fisher8 reorients gradient updates using Fisher geometry, leading to superior uncertainty predictions without introducing complex hyperparameters.
Training neural networks to jointly predict mean and uncertainty estimates from noisy observations can be unstable, prompting a series of independent stabilization efforts. We argue that these interventions highlight a common underlying issue where gradient steps are poorly aligned with the geometry of the loss landscape. To better align updates with local curvature, we derive Fisher8, an output-layer gradient correction that reorients and rescales updates using Fisher geometry rather than Euclidean geometry. Unlike past stabilizers, Fisher8 introduces no data-dependent hyperparameters beyond learning rate and admits an approximate KL trust radius between successive predictive distributions. We show that prior stabilizers converge on overlapping components of this geometric correction. Across multidimensional regression and representation-learning tasks, Fisher8 obtains superior likelihood--error tradeoffs, predicts calibrated uncertainty estimates, and learns rich uncertainty-aware feature spaces.
