Search papers, labs, and topics across Lattice.
This paper introduces CARL, a novel Constraint-Aware Reinforcement Learning framework that enhances the intrinsic constraint awareness of Large Language Models (LLMs) during planning tasks. By implementing a constraint-aware reward mechanism that evaluates output distributions against both constrained and unconstrained inputs, CARL effectively encourages LLMs to prioritize adherence to task constraints. Experimental results across multiple domains, including BlocksWorld and TravelPlanner, show that CARL significantly outperforms traditional Reinforcement Fine-Tuning baselines and state-of-the-art reasoning models, leading to more reliable planning outcomes.
LLMs can now be trained to prioritize task constraints intrinsically, resulting in a dramatic improvement in planning reliability.
Despite their strong reasoning capabilities and extensive world knowledge, Large Language Models (LLMs) frequently generate plans that violate task constraints, undermining their reliability in real-world applications. This deficiency arises from a lack of systematic mechanisms to incorporate constraint information during the generation process. While existing approaches attempt to mitigate this by relying on external tools or task decomposition, they fail to enhance the model's intrinsic constraint awareness. To address this, we propose Constraint-Aware Reinforcement Learning (CARL), a novel RL framework designed to strengthen LLMs' intrinsic focus on constraints. CARL introduces a constraint-aware reward by comparing the model's output distributions under constrained and unconstrained inputs, encouraging constraint focus and penalizing neglect. Compatible with various RL frameworks and requiring no external solvers or top models, CARL enables scalable, end-to-end constraint-aware planning. Extensive experiments on BlocksWorld, TravelPlanner, and T-Eval demonstrate that CARL significantly outperforms standard Reinforcement Fine-Tuning (RFT) baselines and state-of-the-art reasoning models, exhibiting a markedly increased focus on constraints.
