Search papers, labs, and topics across Lattice.
The authors investigate the interface between sparse autoencoders (SAEs) and topic modeling, finding that document-topic mixture estimation and semantic topic characterization fundamentally favor conflicting SAE feature regimes. To resolve this tension, they introduce MonoTM, a decoupled architecture that infers document-topic distributions over full SAE bag-of-features representations before fixing those mixtures to project topics onto a separate, corpus-grounded monosemantic feature vocabulary. Evaluated across three benchmark corpora, this approach preserves global topic geometry while replacing polysemantic top-word lists with far more granular, interpretable conceptual descriptors.
The sparse autoencoder features optimal for document-topic mixture estimation actually undermine topic readability, revealing that monosemantic topic modeling requires decoupling inference from semantic labeling.
Topic models summarize large text corpora, but top-ranked words often provide only a limited representation of topic semantics. Sparse autoencoders (SAEs) offer a way to move beyond word-level descriptors by extracting interpretable features from dense representations, yet how feature interpretability relates to topic-inference quality remains unclear. We introduce \textbf{MonoTM}, an interpretable topic modeling framework that decouples these roles. Across three benchmark corpora, we show that document--topic mixture estimation and semantic interpretation favor different SAE configurations and feature subsets. MonoTM estimates mixtures from the full SAE bag-of-features representation and, with them fixed, learns topic descriptors over a separate vocabulary of corpus-grounded semantic features. This design preserves global topic structure while representing topics with semantic units more meaningful than individual words, making them more useful for downstream corpus analysis.
