Search papers, labs, and topics across Lattice.
This paper investigates the response quality of hybrid-thinking multimodal large language models (MLLMs) by introducing a novel diagnostic benchmark, PatternEval, which evaluates both task accuracy and response-pattern failures. The study reveals that non-thinking inference modes exhibit significantly higher failure rates compared to deliberative thinking, leading to misalignment in response behaviors across different interfaces. By developing PatternRM and PatternRL, the authors demonstrate that incorporating pattern-specific penalties during reinforcement learning can effectively reduce this misalignment while maintaining acceptable task performance levels.
Non-thinking inference in hybrid-thinking MLLMs suffers from widespread response-pattern failures, revealing a critical misalignment that can be mitigated with targeted reinforcement learning strategies.
Hybrid-thinking multimodal large language models (MLLMs) allow a single model to alternate between deliberative thinking and latency-efficient non-thinking inference. Although these modes differ in reasoning budget, their delivered responses should satisfy the same user-facing standard. Correctness alone may not characterize this response quality; we therefore evaluate task accuracy and response-pattern failures as complementary outcomes. We study this gap through \textbf{response-pattern alignment}: whether thinking and non-thinking interfaces preserve acceptable final-response behavior. We introduce \textbf{PatternEval}, a failure-enriched diagnostic benchmark comprising 2,415 multimodal prompts spanning visual perception and grounding, structured image understanding, and multimodal knowledge reasoning. PatternEval tests four recurrent failures: chain-of-thought leakage, response repetition, logical contradiction, and performative reasoning. Response-pattern failures are widespread across models from different providers, with non-thinking inference exhibiting substantially higher failure rates and thereby creating systematic misalignment between thinking and non-thinking interfaces. Motivated by this diagnosis, we develop \textbf{PatternRM}, a response-level reward model, and \textbf{PatternRL}, which introduces pattern-specific penalties during reinforcement learning. Experiments on Qwen3-VL-4B and Qwen3-VL-8B show that incorporating pattern-specific penalties into reinforcement learning can mitigate cross-mode misalignment while incurring a marginal task performance trade-off. Together, PatternEval and PatternRL provide an evaluation-and-training framework for aligning user-visible response patterns across hybrid-thinking interfaces.
