Search papers, labs, and topics across Lattice.
This paper introduces the Planning Experience Exploration and Utilization (PEEU) method, which enhances the task planning capabilities of small multimodal language models by autonomously exploring environments and leveraging hindsight experience to generate high-quality training data. The authors employ a novel Task Decomposition Hierarchical Analysis Framework (TDHAF) to investigate compositional generalization across varying task granularities, revealing that proficiency in low-level skills does not necessarily translate to high-level planning competence. Experimental results show that a 7B model using PEEU achieves 30.6% accuracy, surpassing the larger Qwen2.5-VL-32B model, highlighting the importance of high-level task training for out-of-distribution (OOD) generalization.
Small models can outperform larger counterparts in task planning by leveraging autonomous experience exploration and hindsight training.
Multimodal web agents can assist humans in operating repetitive GUI tasks, where effective task planning is essential for decomposing complex tasks into executable actions. While small open source MLLMs are cost efficient and privacy preserving compared with commercial large models, they suffer from weak planning and limited cross website generalization. To address these limitations, we introduce the planning experience exploration and utilization (PEEU) method, which autonomously explores environments to discover experiences and utilizes hindsight experience to synthesize strictly aligned, high level training data. To quantitatively analyze the generalization behaviors driving this performance, we propose the task decomposition hierarchical analysis framework (TDHAF) to systematically study compositional generalization across three task granularities: low, middle and high levels. Our analysis reveals that mastering low level atomic skills does not guarantee high level planning competence, while high level task training yields stronger OOD generalization. Experiments on real world benchmarks demonstrate PEEU's superior effectiveness: our 7B model achieves 30.6% accuracy, outperforming the much larger Qwen2.5-VL-32B model. These demonstrate constructing hindsight high level tasks and leveraging experiences is crucial for OOD planning abilities of small MLLMs.
