Search papers, labs, and topics across Lattice.
This paper introduces SpectraReward, a novel training-free reward function that leverages pretrained MLLMs to serve as effective reward models for image-generation reinforcement learning. By measuring the log-likelihood of recovering the original prompt from generated images, SpectraReward eliminates the need for preference labels and fine-tuning, while Self-SpectraReward creates a self-improving framework using the model's own understanding for reward assessment. Extensive experiments demonstrate that both approaches significantly enhance generation performance across various models and benchmarks, challenging the assumption that larger reward models are always superior.
SpectraReward reveals that pretrained MLLMs can serve as powerful zero-shot reward models, outperforming traditional methods without the need for fine-tuning or external labels.
In this paper, we propose SpectraReward, a training-free reward function that turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning. Instead of asking the MLLM to judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass. We use the average image-conditioned prompt log-likelihood as the reward, directly reusing the MLLM's pretrained image-text alignment ability without preference labels, reward-model fine-tuning. We further introduce Self-SpectraReward, a special case for unified multimodal models where the policy's own understanding branch serves as the reward model for its generation branch, forming a closed-loop self-improving framework without external reward models or external knowledge. Extensive experiments validate SpectraReward through a broad image-generation RL study covering two diffusion models, three RL algorithms, nine reward MLLM backbones from four MLLM families spanning 4B to 235B parameters, and five out-of-distribution text-to-image benchmarks. Results show that both SpectraReward and Self-SpectraReward significantly and consistently improve generation performance and outperform prior MLLM-derived reward training methods. Further analysis reveals that larger reward MLLMs are not always better, while Self-SpectraReward can match or surpass much larger external reward models, suggesting that reward-policy alignment is a key factor for effective image-generation RL. Project Page: https://huangrh99.github.io/SpectraReward/
