Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To prevent sparse autoencoders from wasting latent capacity on low-level lexical and formatting artifacts, this work introduces chunk-level SAEs trained on mean-pooled activations over contiguous token spans using intra-chunk reconstruction, cross-chunk prediction, and joint objectives. Moving the autoencoding objective to multi-token chunks decouples semantic feature discovery from token-level surface noise while preserving interpretability at matched compute. Across downstream benchmarks, Mean-Chunk SAEs yield superior concept steering and reasoning detection, while predictive Cross-Chunk variants excel at document retrieval and zero-shot transfer classification.
Standard token-level SAEs squander their sparsity budgets memorizing punctuation and syntax; training on pooled multi-token chunks forces dictionaries to capture true, steerable semantic concepts and latent reasoning.
Sparse autoencoders (SAEs) expose features that help us understand and steer language models, but faithful reconstruction does not guarantee informative concepts. Token-level objectives reward lexical and formatting details alongside semantic content, all competing for a limited sparse budget. We introduce a family of chunk-level SAEs that encode mean-pooled activations over chunks, each a contiguous span of tokens: Mean-Chunk reconstructs the observed chunk, Cross-Chunk predicts an independently processed neighbor, and Joint-Chunk combines both targets. These designs separate the effect of a larger observation unit from that of predicting information shared across passages. With matched training data, chunk-level SAEs remain powerful interpretability tools while learning reliable semantic features that capture high-level concepts and respond selectively to relevant content. Their strengths are complementary: Mean-Chunk improves high-level feature discovery, reasoning detection beyond surface cues, and steering; Cross-Chunk leads document retrieval and classification transfer while producing selective, persistent features. Changing what an SAE sees and predicts yields reliable semantic features for more meaningful tasks. We demonstrate their practical value through gains across downstream tasks such as retrieval, reasoning detection, and steering.