Search papers, labs, and topics across Lattice.
This paper introduces NormGuard, a hinge penalty designed to suppress norm inflation in flow-matching reinforcement learning, which typically occurs during post-training fine-tuning and negatively impacts perceptual quality. The authors demonstrate that while traditional inference-time norm rescaling fails to improve reward or quality, their training-time intervention effectively mitigates quality degradation without sacrificing reward alignment. Experimental results across multiple models and post-training methods reveal that NormGuard enhances image quality and realism, particularly during few-step inference, while maintaining reward integrity.
NormGuard effectively preserves reward alignment in RL fine-tuning while significantly enhancing perceptual quality, challenging the notion that post-training improvements come at the cost of visual fidelity.
Reinforcement learning (RL) post-training improves the reward alignment of flow-based generators, but often degrades perceptual quality in ways that are not captured by the reward proxy. We identify a simple structural signature of this drift: across three post-training methods (NFT, AWM, DPO), RL fine-tuning inflates the per-step velocity norm |v_胃| by 5% to 15% relative to the reference. A form of norm inflation has been studied in classifier-free guidance (CFG), where rescaling the velocity back to a reference norm at inference time can mitigate the resulting artifacts. However, this inference-time correction does not transfer cleanly to RL: rescaling v_胃 to match |v_{ref}| at inference time neither improves reward nor fixes the quality degradation, because the inflation is co-adapted into the model weights. Furthermore, an adjoint sensitivity analysis shows that velocity magnitude rescaling carries no coherent first-order reward signal at the batch level, indicating that suppressing norm inflation is unlikely to remove a consistently reward-carrying component. Since inference-time renormalization fails while norm suppression carries no reward cost, training-time intervention is the appropriate strategy. Together, these findings motivate \methodname, a hinge penalty that activates only when |v_胃| exceeds |v_{ref}| and composes additively with any velocity-local base loss. Across two base models, three post-training methods, and two reward proxies, \methodname consistently improves MLLM-judged image quality and forensic realism while preserving reward, with gains that amplify under few-step inference and are not explained by early stopping.
