Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors formalize "effective interference," an analytical framework that integrates activation statistics and sign dynamics with dictionary geometry to distinguish constructive from destructive feature interactions in concept extraction. Under local fixed-support assumptions, they theoretically decompose how architectural levers—specifically feature orthogonalization, bias compensation, gain adaptation, and encoder-decoder asymmetry—govern these interactions. Empirically evaluating sparse autoencoders, they demonstrate that constrained architectures selectively suppress interference among co-active features while leveraging decoupled encoders, bias, and gain to preserve beneficial cross-talk.
Geometric overlap alone misrepresents feature interference in sparse autoencoders; realized cross-talk is heavily governed by activation statistics and actively steered by architectural levers like bias compensation and encoder-decoder decoupling.
Interference is commonly treated as geometric overlap between learned features. We introduce effective interference, which combines feature geometry and code statistics to capture realized interactions, distinguishing constructive from destructive interference and frequent weak interactions from rare strong ones. Under local fixed-support assumptions, we characterize how architectural constraints shape interference through four mechanisms: feature orthogonalization, bias compensation, gain adaptation, and encoder-decoder separation. Experiments with sparse autoencoders show that constrained architectures selectively reduce overlap among co-active features, while bias, gain, and encoder freedom allow constructive cross-contributions to remain. Together, these results show that interference in learned representations depends not only on feature geometry, but also on how features are used and on the architecture that produces their codes.