Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates whether training biases in minimal MLPs can induce hidden neuron specialization and improve training data reconstruction from learned weights. They compare coverage, separation, and overlap losses against a standard fitting baseline on 1D datasets, finding that coverage regularization minimizes reconstruction error and enhances prototype specialization. The study reveals that repulsive structural losses, like overlap penalties, can lead to degenerate equilibria where prototype centers are pushed outside the training data's convex hull, unless compensated by an attractor.
Repulsive loss terms in minimal MLPs can backfire, collapsing the latent space geometry unless balanced by attractive forces.
We here study whether training biases can make hidden neurons specialize in minimal one-hidden-layer MLPs, and whether such specialization improves prototype-based reconstruction of the training dataset from the learned weights. We consider Gaussianactivation MLPs of width equal to dataset size and compare three structural losses that respectively encourage coverage of the training samples, separation between neuron-induced prototypes, and low overlap of hidden responses, against the standard fitting baseline. Experiments on uniformly sampled one-dimensional datasets show a stable pattern from N = 3 to N = 100 across 480 controlled runs. Coverage regularization gives the lowest mean reconstruction error at every tested size and raises the prototype-usage specialization ratio relative to the standard baseline, while separation has mixed effects and overlap penalties are systematically harmful. We show that the harm is not an optimization failure: overlap-active approaches fit the data as well as overlap-free ones but route the optimizer to a degenerate equilibrium in which prototype centers are pushed outside the convex hull of the training inputs. Coverage cannot reward this expulsion and acts as an attractor: separation admits it only at large temperature and overlap admits it at the nominal hyperparameter choice. A direct τ-sweep on the separation-only mask and a prototype-position visualization at N = 100 confirm the mechanism. The findings yield a simple design principle for prototype-recoverability-aware training: every repulsive structural loss must be compensated by a compatible attractor, or it will collapse the latent geometry it was meant to refine.