Search papers, labs, and topics across Lattice.
This paper introduces a Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoder (SAE) to extract cross-seed universal features from independently trained BERT models, addressing the challenge of misaligned feature spaces due to non-convex dictionary learning. By applying an orthogonal Procrustes rotation to align activation spaces before joint training, the authors enhance feature universality and interpretability. The method outperforms post-hoc alignment baselines, achieving Pearson correlation coefficients of at least 0.70 across multiple datasets, while qualitatively revealing interpretable sociolinguistic patterns in the features extracted.
Achieving over 70% correlation in cross-seed feature universality reveals that BERT models can be aligned to uncover interpretable sociolinguistic insights.
We present a Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoder (SAE) for extracting cross-seed universal features from independently trained BERT models. Cross-seed feature universality is a fundamental challenge in mechanistic interpretability: because dictionary learning is non-convex, independently trained networks learn misaligned feature spaces, so apparently identical features may differ by random initialization. We address this by computing an orthogonal Procrustes rotation between seeds'activation spaces before joint SAE training, combining Top-K sparsity, end-to-end downstream optimization, and an auxiliary dead-feature revival loss based on previous SAE literature. Evaluating on five independent seed pairs (ten BERT models) across three benchmark datasets (SST-2, Stanford Politeness, TweetEval Emotion), our full pipeline produces more universal features (Pearson r $\geq$ 0.70 across seeds) than post-hoc alignment baselines on all three datasets. A minimal qualitative analysis confirms that high-universality features encode interpretable sociolinguistic patterns.
