Search papers, labs, and topics across Lattice.
The paper introduces SVD-Attention, a novel attention mechanism that leverages the low-rank structure prevalent in recommender systems to reduce the computational complexity of attention from $O(N^2 d)$ to $O(Ndr)$ while preserving the softmax operation. Based on SVD-Attention, they propose SOLAR, a sequence modeling framework for recommendation that handles long behavior sequences and large candidate sets. Evaluated in Kuaishou's online recommendation scenario, SOLAR achieved a 0.68% increase in Video Views, demonstrating its practical effectiveness.
SVD-Attention slashes the quadratic cost of attention to linear for recommendation tasks by exploiting the inherent low-rank structure of user behavior sequences, without sacrificing softmax.
Attention mechanism remains the defining operator in Transformers since it provides expressive global credit assignment, yet its $O(N^2 d)$ time and memory cost in sequence length $N$ makes long-context modeling expensive and often forces truncation or other heuristics. Linear attention reduces complexity to $O(N d^2)$ by reordering computation through kernel feature maps, but this reformulation drops the softmax mechanism and shifts the attention score distribution. In recommender systems, low-rank structure in matrices is not a rare case, but rather the default inductive bias in its representation learning, particularly explicit in the user behavior sequence modeling. Leveraging this structure, we introduce SVD-Attention, which is theoretically lossless on low-rank matrices and preserves softmax while reducing attention complexity from $O(N^2 d)$ to $O(Ndr)$. With SVD-Attention, we propose SOLAR, SVD-Optimized Lifelong Attention for Recommendation, a sequence modeling framework that supports behavior sequences of ten-thousand scale and candidate sets of several thousand items in cascading process without any filtering. In Kuaishou's online recommendation scenario, SOLAR delivers a 0.68\% Video Views gain together with additional business metrics improvements.
