Search papers, labs, and topics across Lattice.
This study investigates the emergence of compositionality in neural networks, revealing that it is contingent upon a specific balance of depth and connectivity. The authors find that compositional structures only arise in certain sparse networks and at particular depths, with deviations leading to non-compositional, fractured solutions. By introducing similarity-based pruning and a heuristic depth predictor, they provide a method to enhance compositionality in model architectures, supported by a theoretical framework that elucidates the conditions necessary for its emergence.
Compositionality in neural networks only emerges in a narrow depth-connectivity regime, with specific architectural constraints dictating success or failure.
Compositionality is believed to be the foundation for generalization, enabling models to reuse meaningful primitives in novel combinations. Yet, models trained with standard gradient-based optimization rarely, and often only weakly, exhibit compositional internal structure, and it remains unclear how or why such compositionality forms. In this work, we show that compositionality emerges in a narrow connectivity-depth sweet spot. Along the connectivity axis, compositionality only appears in some specifically sparse networks, heavily depends on which connections remain rather than on weights' sparsity alone. Along the depth axis, compositionality emerges within a narrow, target-dependent regime, peaking at specific depths, while both shallower and deeper networks fail. When either the depth or connectivity condition is violated, gradient descent silently converges to fractured solutions rather than compositional ones. To discover and exploit this emergence, we introduce (i) similarity-based pruning (SP) to recover compositional connectivity and (ii) a heuristic depth predictor to estimate where compositionality is most likely to appear. Finally, we support these empirical findings with a theoretical framework based on compositional sparsity, volume-ratio arguments, and feature-interference bounds, explaining why compositional solutions are reachable only in a narrow depth-connectivity regime.
NUS