Search papers, labs, and topics across Lattice.
This paper investigates the curvature of loss surfaces in Probabilistic Circuits (PCs) by decomposing the Hessian trace into contributions from circuit flow and local sharpness terms. The authors demonstrate that traditional global sharpness regularization can mislead learning in PCs due to their compositional nature, leading to underfitting. They introduce an adaptive sharpness-aware regularizer that effectively balances local curvature penalties with closed-form EM updates, resulting in improved generalization without sacrificing robustness.
Global sharpness regularization can lead to underfitting in Probabilistic Circuits, but a new adaptive approach recovers generalization while maintaining robustness.
Probabilistic Circuits (PCs) are generative models that support exact inference and, unlike deep neural networks, admit an exact and tractable measure of loss-surface curvature: the trace of the Hessian of the log-likelihood. Recent work regularizes this trace globally to bias learning toward flatter, better generalizing optima. We show that treating sharpness as a global regularizer can be misspecified for PCs, whose curvature is inherently compositional. We prove that each sum node's contribution to the Hessian trace factorizes exactly into its circuit flow, which measures how heavily the node is used, and a local sharpness term determined by its output distribution. This decomposition provides insights into why global sharpness regularization is depth biased and can lead to underfitting. Building on it, we introduce an adaptive sharpness aware regularizer that penalizes nodes based on intrinsic local curvature and preserves closed form EM updates. We also show that empirically, this targeted regularization recovers the generalization that global regularization sacrifices while retaining the robustness and benefits of sharpness aware learning.
