Search papers, labs, and topics across Lattice.
This paper introduces Structure-Aware Fine-Tuning (SAFT), a self-supervised method that enhances the reliability of reward signals from Vision-Language Models (VLMs) in Reinforcement Learning (RL) by leveraging intrinsic structural priors. The authors demonstrate that SAFT effectively denoises the reward landscape, leading to faster policy convergence and improved alignment, as measured by EPIC distance, across various base model capabilities. This approach not only mitigates the challenges of noisy reward signals but also highlights the importance of structural inductive biases in stabilizing text-conditioned RL without the need for extensive human annotation.
By refining VLM-derived reward signals with structural priors, SAFT transforms noisy feedback into a reliable guide for faster and more aligned policy learning.
Designing effective reward functions remains a major bottleneck in Reinforcement Learning (RL). Recent work uses large foundation Vision-Language Models (VLMs) as reward models, computing text-observation similarity to bypass manual reward engineering. Although promising, these rewards are often noisy and unreliable, limiting their direct utility during deployment. We present Structure-Aware Fine-Tuning (SAFT), a simple, self-supervised method that refines these imperfect reward signals online without access to ground-truth supervision. SAFT leverages intrinsic structural priors to regularize the VLM's latent space via LoRA adapters. We rigorously evaluate SAFT across a spectrum of base model capabilities to demonstrate its versatility. Our results show that SAFT consistently denoises the reward landscape, yielding faster policy convergence and substantially improved alignment (EPIC distance) relative to the underlying base model, suggesting that failures can often be attributed to structural brittleness rather than semantic misunderstanding. By replacing extensive human preference annotation with structural inductive biases inherent to the task, SAFT offers a scalable path for stabilizing text-conditioned RL and underscores the broader value of incorporating task structure as a general inductive bias.
ETH