Search papers, labs, and topics across Lattice.
The paper investigates the robustness and chain-of-thought (CoT) consistency of reinforcement learning (RL) fine-tuned vision-language models (VLMs) when subjected to textual perturbations such as misleading captions or incorrect CoT traces. It demonstrates that these perturbations significantly reduce model robustness and confidence, particularly when CoT consistency is considered, and reveals an accuracy-faithfulness trade-off during RL fine-tuning, where improved accuracy can degrade the reliability of the CoT. The study finds that while adversarial augmentation improves robustness, it doesn't prevent faithfulness drift, and incorporating a faithfulness-aware reward can restore alignment but risks shortcut strategies when combined with augmentation.
RL fine-tuning can make vision-language models *less* reliable reasoners, as gains in benchmark accuracy come at the cost of faithfulness to the underlying visual grounding and chain-of-thought.
Reinforcement learning (RL) fine-tuning has become a key technique for enhancing large language models (LLMs) on reasoning-intensive tasks, motivating its extension to vision language models (VLMs). While RL-tuned VLMs improve on visual reasoning benchmarks, they remain vulnerable to weak visual grounding, hallucinations, and over-reliance on textual cues. We show that simple, controlled textual perturbations--misleading captions or incorrect chain-of-thought (CoT) traces--cause substantial drops in robustness and confidence, and that these effects are more pronounced when CoT consistency is taken into account across open-source multimodal reasoning models. Entropy-based metrics further show that these perturbations reshape model uncertainty and probability mass on the correct option, exposing model-specific trends in miscalibration. To better understand these vulnerabilities, we further analyze RL fine-tuning dynamics and uncover an accuracy-faithfulness trade-off: fine-tuning raises benchmark accuracy, but can simultaneously erode the reliability of the accompanying CoT and its robustness to contextual shifts. Although adversarial augmentation improves robustness, it does not by itself prevent faithfulness drift. Incorporating a faithfulness-aware reward can restore alignment between answers and reasoning, but when paired with augmentation, training risks collapsing onto shortcut strategies and robustness remains elusive. Together, these findings highlight the limitations of accuracy-only evaluations and motivate training and assessment protocols that jointly emphasize correctness, robustness, and the faithfulness of visually grounded reasoning.
Apple ML