Search papers, labs, and topics across Lattice.
The paper addresses the challenge of applying LLMs to high-frequency decision-making tasks, where existing methods struggle due to the sensitivity to small numerical state fluctuations and policy misalignment between sub-tasks and composite tasks. They introduce Normalized Action Reward-guided Consistency Policy Optimization (NAR-CP), which normalizes action rewards to improve policy learning and uses a consistency loss to align global semantic policies with sub-semantic policies. Experiments on UAV pursuit demonstrate that NAR-CP achieves superior performance and generalization in both independent and composite high-frequency tasks.
LLMs can now excel in high-frequency decision-making tasks like UAV pursuit, thanks to a novel reward normalization and consistency loss approach that aligns global and sub-semantic policies.
While Large Language Models (LLMs) form the cornerstone of sequential decision-making agent development, they have inherent limitations in high-frequency decision tasks. Existing research mainly focuses on discrete embodied decision scenarios with low-frequency and significant semantic differences in state space (e.g., household planning). These methods suffer from limited performance in high-frequency decision-making tasks, since high-precision numerical state information in such tasks undergoes frequent updates with minimal fluctuations, and exhibiting policy misalignment between the learned sub-tasks and composite tasks. To address these issues, this paper proposes Normalized Action Reward guided Consistency Policy Optimization (NAR-CP). 1) Our method first acquires predefined dense rewards from environmental feedback of candidate actions via reward functions, then completes reward shaping through normalization, and theoretically verifies action reward normalization does not impair optimal policy. 2) To reduce policy misalignment in composite tasks, we use LLMs to infer sub-observation candidate actions and generate joint policies, with consistency loss ensuring precise alignment between global semantic policies and sub-semantic policies. Experiments on UAV pursuit, a typical high-frequency task, show our method delivers superior performance on independent and composite tasks with excellent generalization to unseen tasks.
