Search papers, labs, and topics across Lattice.
This study investigates the response quality of hybrid-thinking multimodal large language models (MLLMs) by evaluating both task accuracy and response-pattern failures through a novel benchmark called PatternEval. The findings reveal that non-thinking inference modes exhibit significantly higher rates of response-pattern failures compared to deliberative thinking, leading to misalignment in user-facing responses. To address this issue, the authors propose PatternRM and PatternRL, which incorporate pattern-specific penalties during reinforcement learning, successfully aligning responses across different inference modes with only a minor trade-off in task performance.
Non-thinking inference in hybrid-thinking MLLMs suffers from a staggering increase in response-pattern failures, revealing a critical misalignment that can undermine user trust.
Hybrid-thinking multimodal large language models (MLLMs) allow a single model to alternate between deliberative thinking and latency-efficient non-thinking inference. Although these modes differ in reasoning budget, their delivered responses should satisfy the same user-facing standard. Correctness alone may not characterize this response quality; we therefore evaluate task accuracy and response-pattern failures as complementary outcomes. We study this gap through response-pattern alignment: whether thinking and non-thinking interfaces preserve acceptable final-response behavior. We introduce PatternEval, a failure-enriched diagnostic benchmark comprising 2,415 multimodal prompts spanning visual perception and grounding, structured image understanding, and multimodal knowledge reasoning. PatternEval tests four recurrent failures: chain-of-thought leakage, response repetition, logical contradiction, and performative reasoning. Response-pattern failures are widespread across models from different providers, with non-thinking inference exhibiting substantially higher failure rates and thereby creating systematic misalignment between thinking and non-thinking interfaces. Motivated by this diagnosis, we develop PatternRM, a response-level reward model, and PatternRL, which introduces pattern-specific penalties during reinforcement learning. Experiments on Qwen3-VL-4B and Qwen3-VL-8B show that incorporating pattern-specific penalties into reinforcement learning can mitigate cross-mode misalignment while incurring a marginal task performance trade-off. Together, PatternEval and PatternRL provide an evaluation-and-training framework for aligning user-visible response patterns across hybrid-thinking interfaces.
