Search papers, labs, and topics across Lattice.
The paper introduces RankUp, a novel recommender system architecture designed to address representation collapse in deep MetaFormer-based models. RankUp employs techniques like randomized permutation splitting, multi-embedding paradigms, and global token integration to enhance the effective rank of token representations across layers. Deployed in Weixin Video Accounts, Official Accounts, and Moments, RankUp achieved significant GMV improvements of 3.41%, 4.81%, and 2.21%, respectively, demonstrating its effectiveness in large-scale production environments.
RankUp tackles representation collapse in deep recommender systems, unlocking significant GMV gains in real-world deployments by strategically boosting the effective rank of token representations.
The scaling laws for recommender systems have been increasingly validated, where MetaFormer-based architectures consistently benefit from increased model depth, hidden dimensionality, and user behavior sequence length. However, whether representation capacity scales proportionally with parameter growth remains largely unexplored. Prior studies on RankMixer reveal that the effective rank of token representations exhibits a damped oscillatory trajectory across layers, failing to increase consistently with depth and even degrading in deeper layers. Motivated by this observation, we propose \textbf{RankUp}, an architecture designed to mitigate representation collapse and enhance expressive capacity through randomized permutation splitting over sparse features, a multi-embedding paradigm, global token integration, crossed pretrained embedding tokens and task-specific token decoupling. RankUp has been fully deployed in large-scale production across Weixin Video Accounts, Official Accounts and Moments, yielding GMV improvements of 3.41\%, 4.81\% and 2.21\%, respectively.
