Search papers, labs, and topics across Lattice.
This paper investigates the application of FP8 post-training quantization to the OneRec-V2 generative recommendation model, highlighting its more controlled weight and activation statistics compared to traditional recommendation models. They develop an FP8 quantization framework integrated with optimized inference infrastructure, achieving a 49% reduction in end-to-end inference latency and a 92% increase in throughput. Online A/B testing validates that FP8 inference maintains performance, suggesting LLM optimization techniques can be adapted to modern recommendation systems.
Generative recommendation models like OneRec-V2 can achieve near-lossless FP8 quantization, unlocking significant latency and throughput improvements, unlike traditional recommender systems.
Quantized inference has demonstrated substantial system-level benefits in large language models while preserving model quality. In contrast, reliably applying low-precision quantization to recommender systems remains challenging in industrial settings. This difficulty arises from differences in training paradigms, architectural patterns, and computational characteristics, which lead to distinct numerical behaviors in weights and activations. Traditional recommender models often exhibit high-magnitude and high-variance weights and activations, making them more sensitive to quantization-induced perturbations. In addition, recommendation workloads frequently suffer from limited hardware utilization, limiting the practical gains of low-precision computation. In this work, we revisit low-precision inference in the context of generative recommendation. Through empirical distribution analysis, we show that the weight and activation statistics of OneRec-V2 are significantly more controlled and closer to those of large language models than traditional recommendation models. Moreover, OneRec-V2 exhibits a more compute-intensive inference pattern with substantially higher hardware utilization, enabling more end-to-end throughput gains with low-precision computation. Leveraging this property, we develop a FP8 post training quantization framework and integrate it into an optimized inference infrastructure. The proposed joint optimization achieves a 49\% reduction in end-to-end inference latency and a 92\% increase in throughput. Extensive online A/B testing further confirms that FP8 inference introduces no degradation in core metrics. These results suggest that as recommender systems evolve toward the paradigms of large language models, algorithm-level and system-level optimization techniques established in the LLM domain can be effectively adapted to large-scale recommendation workloads.
