Search papers, labs, and topics across Lattice.
This paper introduces LEACL, a novel framework that leverages large language models (LLMs) to enhance automatic curriculum learning (ACL) for long-horizon manipulation tasks in reinforcement learning. By utilizing LLMs to decompose complex tasks into subtasks and generate task-dependent specifications, LEACL effectively guides learning using only sparse reward signals, thereby reducing reliance on manually crafted dense rewards. The evaluation on five long-horizon tasks from the LIBERO benchmark demonstrates that LEACL outperforms traditional methods, achieving superior success rates without the biases introduced by dense reward design.
LLMs can transform sparse reward signals into structured learning paths, leading to better performance in complex manipulation tasks than traditional dense reward systems.
Long-horizon manipulation tasks pose significant challenges for reinforcement learning due to sparse reward signals and long horizons. Automatic curriculum learning (ACL) has been proposed to tackle these challenges by progressively training agents on a sequence of tasks, from easier to more difficult. However, the success of ACL depends heavily on task-dependent specifications-such as well-defined task parameter spaces and difficulty measures-which are often manually crafted and difficult to generalize across diverse tasks. Recent advances in large language models (LLMs) offer a promising alternative by enabling the decomposition of complex tasks into meaningful subtasks using the LLMs'web-scale common-sense knowledge. This decomposition can provide a natural curriculum structure for efficient learning of long-horizon tasks. However, existing LLM-based methods typically rely on hand-designed dense reward functions to learn each subtask, which can introduce bias and still requires significant human supervision. In this work, we propose LLM-enhanced automatic curriculum learning (LEACL), a framework that integrates LLMs and ACL to address these limitations. Specifically, LLMs are used to both decompose tasks into subtasks and to generate task-dependent specifications for each subtask. These specifications are then used by ACL algorithms to guide learning using only sparse reward signals, eliminating the need for dense reward design. We evaluate LEACL on five long-horizon manipulation tasks from the LIBERO benchmark. LEACL achieves better asymptotic performance in terms of the success rates compared to human-designed dense rewards.
