Search papers, labs, and topics across Lattice.
This paper identifies and addresses the performance bottleneck in standard Transformer architectures for recommendation systems, which is caused by severe embedding and attention collapse due to the unique characteristics of recommendation data. The authors introduce SpecFormer, a Spectral-Aware Transformer that incorporates a Learnable Spectral Softening module, a Spectrum-softened Attention mechanism, and Spectral Residual Position Encoding to mitigate these issues. Extensive experiments show that SpecFormer not only outperforms existing models but also scales effectively in real-world applications, enhancing both attention rank and recommendation performance.
SpecFormer transforms recommendation systems by mitigating embedding and attention collapse, leading to superior performance and scalability.
Transformer architectures have achieved remarkable success across diverse domains; however, directly applying their standard self-attention mechanism to recommendation often yields suboptimal performance, sometimes even trailing behind well-designed simple recommendation models. In this paper, we reveal that this performance bottleneck stems from severe embedding and attention collapse unique to recommendation scenarios. The heterogeneity and long-tail nature of recommendation data lead to a severe spectral collapse dominated by a few principal singular values. We further theoretically demonstrate that this triggers a vicious cycle in recommendation model's forward and backward propagation, which accelerates embedding and attention collapse and limits the model's scaling capability with increased depth. To address these issues, we propose SpecFormer, a novel Spectral-Aware Transformer designed for mitigating embedding and attention collapse in recommendation. Specifically, SpecFormer introduces 1) a Learnable Spectral Softening module to dynamically smooth the singular values distribution of the input token embeddings; 2) a Spectrum-softened Attention mechanism to model feature interaction under a more uniform spectral distribution space; 3) a Spectral Residual Position Encoding via Taylor expansion of singular values, explicitly providing a spectral inductive bias for feature interactions. Extensive experiments on one industrial and two public datasets demonstrate that SpecFormer significantly outperforms state-of-the-art baselines. Notably, SpecFormer has been successfully deployed in a real-world commercial recommender system and exhibits exceptional scaling capabilities: stacking SpecFormer layers actively improves the attention effective rank and recommendation performance.
