Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Rather than treating personalized alignment as a generator capability bottleneck, this work frames personalized generation as a test-time candidate-matching problem solved via lightweight, million-parameter ranking heads. By directly reusing the base generator's internal embeddings and training on fine-grained user preferences, the framework bypasses the prohibitive latency and poor personalization calibration typical of monolithic reward models. Across nine benchmarks spanning three personalization domains, these factorized micro-rankers consistently outperform billion-parameter reward models using under 0.4% of their parameters while slashing scoring latency by four orders of magnitude.
Personalized alignment is fundamentally a candidate-matching problem rather than a generator bottleneck: million-parameter MLPs tapping internal LLM embeddings beat monolithic, billion-parameter reward models across the board at 10,000× lower latency.
Aligning large language models (LLMs) to diverse user preferences is fundamentally hindered by standard alignment paradigms that optimize for monolithic users. In this work, empirical studies are first used to reveal the existence of a massive, untapped performance headroom for personalized generation through test-time alignment. We demonstrate that personalized generation is uniquely suited for test-time scaling methods like Best-of-N (BoN) because it can be viewed primarily as a candidate matching problem rather than a generator capability bottleneck. While reward models could in principle exploit this headroom, they are poorly calibrated for personalization, and their billion-parameter scale makes scoring large candidate pools prohibitively expensive. To overcome this limitation, we propose a parameter-efficient framework utilizing million-parameter scale multi-layer perceptron (MLP) ranking models. Our personalized ranking model directly reuses the internal embeddings of the base generator with minimal overhead. By scaling train-time data to provide fine-grained personalized preferences, this million-parameter ranking model accurately scores large candidate pools and can seamlessly guide generation to reduce the cost of materializing N candidates. Extensive experiments on nine datasets spanning three personalized generation settings show that our personalized ranking model effectively exploits the discovered headroom, outperforming billion-parameter generalist reward models on every dataset, with under 0.4% of their parameters and four orders of magnitude lower scoring latency.