Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces the quadrilateral loss, a novel differentiable penalty designed to measure and enforce additivity in dense neural networks by evaluating second-order mixed differences on training point pairs. The findings reveal that this approach allows for the removal of learned interactions with minimal cost, enhancing both model accuracy and interpretability on small datasets. Additionally, the study uncovers that pre-regularization interaction magnitudes do not reliably predict the interactions retained post-regularization, challenging traditional post-hoc interaction rankings.
The quadrilateral loss reveals that learned interactions can be removed with minimal impact on performance, fundamentally altering our understanding of model interpretability.
Additive models buy interpretability by forbidding feature interactions, a constraint that neural instantiations enforce architecturally. We introduce the quadrilateral loss, a differentiable penalty that treats additivity as a measurable behavior instead: a second-order mixed difference on pairs of training points swapping one coordinate, which vanishes if and only if the coordinate carries no interaction, remains informative for piecewise-linear networks, and equals in expectation the per-coordinate interaction mass of the interventional Shapley-GAM. The loss turns additivity into a dial - most learned interactions prove removable almost for free, and on small datasets a moderate penalty improves accuracy and additivity simultaneously - and into an online observable: its per-feature surrender curves show, across seeds and datasets, that pre-regularization interaction magnitude barely predicts what a regularized model retains, undermining post-hoc interaction rankings. Against this instrument we compare routes to exact additivity, spanning structural masks, behavioral penalties (optionally crystallized into exact structure), weight decay, backfitting, the shared-section model, and bagged boosted stumps: constraining behavior before structure dominates weight-space constraints, rankings reverse between data regimes, and converging routes agree on the shape functions themselves. Three silent failure modes we document share one anatomy: guarantees imported into settings that quietly void their preconditions.