Search papers, labs, and topics across Lattice.
This paper addresses the "Thinking-Acting Gap" in agentic vision-language models, where tool use is infrequent and often incorrect during reinforcement learning from preference optimization (RLPO). They introduce Agent eXplorative Policy Optimization (AXPO), which resamples tool calls and continuations within subgroups of all-wrong tool-using rollouts, conditioned on a fixed thinking prefix. Experiments across nine multimodal benchmarks show that AXPO significantly improves performance over standard GRPO, achieving better results with smaller models.
A simple resampling strategy closes the "Thinking-Acting Gap" in agentic VLMs, enabling smaller models to outperform larger ones on multimodal reasoning tasks.
Vision-language models with extended reasoning succeed on complex problems, but many real-world problems require external tools that internal reasoning alone often cannot resolve. Agentic reasoning therefore interleaves two behaviors with a structural asymmetry: thinking (the self-contained default) and tool use (a high-variance auxiliary acting). We refer to this asymmetry as the Thinking-Acting Gap. Under standard RL recipes like GRPO, the gap manifests as two diagnostic symptoms during training: tool use is attempted on only ~30% of rollouts, and when attempted, the tool-using rollouts within a group are all-wrong on ~40% of questions, suppressing the learning signal at the tool calls that needed it. We propose AXPO (Agent eXplorative Policy Optimization): for each all-wrong tool-using subgroup, AXPO fixes the thinking prefix and resamples the tool call and its continuation, paired with uncertainty-based prefix selection. Across nine multimodal benchmarks and three scales of Qwen3-VL-Thinking, SFT+AXPO outperforms SFT+GRPO at average (+1.8pp Pass@1 and +1.8pp Pass@4 at 8B on average) and 8B with SFT+AXPO surpasses the 32B Base on Pass@4 with 4 times fewer parameters.
