Search papers, labs, and topics across Lattice.
This paper establishes a connection between singular learning theory and information geometry by introducing the concept of "dead directions," which are unit vectors where the Fisher metric degenerates. The authors demonstrate that the KL order of these dead directions can be recovered from the decay rate of the directional Fisher curvature, allowing for a deeper understanding of singularities in overparameterized models. Additionally, they propose a new preconditioner, DDCAdam, which effectively utilizes this framework to enhance optimization in deep networks, providing a practical tool for researchers working with complex architectures.
The discovery of "dead directions" reveals a pathway to recover KL divergence rates in overparameterized models, transforming our approach to singular learning in deep networks.
Singular learning theory and information geometry have studied the same parameter spaces in mostly separate vocabularies: the former computes Bayesian invariants in resolved coordinates, the latter works in original coordinates under a non-degeneracy assumption that overparameterised models routinely violate. We bridge them through one primitive, the dead direction: a unit vector along which the Fisher metric degenerates, equivalently a tangent to the analytic singular set with a definite KL order, set by how fast the KL divergence vanishes. The two readings name the same vector; our central move shows its KL order is recoverable as the decay rate of the directional Fisher curvature approaching the singularity, in original parameter coordinates and without a Hironaka resolution. A selection rule on smooth fibres translates this rate into Watanabe's single-direction contribution to the real log canonical threshold, and we extend the recovery to multi-component crossings, multiplicity $m$, the singular fluctuation $谓$ (universal in the KL order for 1D directions), prior-RLCT shifts, and tempered posteriors. We then lift this rate to a deep network: a multi-layer K-FAC factorisation writes each Fisher block as a product of activation- and gradient-side rates with a duality between them, instantiated at modern-network primitives (residual streams, layer normalisation, attention). A quotient theorem carries the rate to the gauge quotient $螛/G$ under gradient flow on a $G$-invariant metric; SGD qualifies, standard Adam does not, and we construct a $G$-equivariant Adam-family preconditioner (DDCAdam) that does. The bridge yields a parameter-coordinate handle on singular geometry, closed-form per-architecture predictions, and a trajectory-rate readout of Watanabe's triple $(位, m, 谓)$ from one checkpoint's forward and backward passes, without posterior sampling.
