Search papers, labs, and topics across Lattice.
The paper introduces Multi-Agent Constitutional Learning (MAC), a novel approach to automatically learning constitutions for LLMs by optimizing structured prompts represented as sets of rules. MAC employs a network of specialized agents to accept, edit, or reject rule updates, and is further enhanced by MAC+, which trains agents on successful trajectories. Experiments on PII tagging and agentic tool calling show that MAC outperforms existing prompt optimization methods by over 50%, generates interpretable rule sets, and matches the performance of supervised fine-tuning and GRPO without parameter updates.
Forget hand-crafted rules: MAC learns interpretable LLM constitutions that beat prompt optimization by 50% and rival fine-tuning, all without parameter updates.
Constitutional AI is a method to oversee and control LLMs based on a set of rules written in natural language. These rules are typically written by human experts, but could in principle be learned automatically given sufficient training data for the desired behavior. Existing LLM-based prompt optimizers attempt this but are ineffective at learning constitutions since (i) they require many labeled examples and (ii) lack structure in the optimized prompts, leading to diminishing improvements as prompt size grows. To address these limitations, we propose Multi-Agent Constitutional Learning (MAC), which optimizes over structured prompts represented as sets of rules using a network of agents with specialized tasks to accept, edit, or reject rule updates. We also present MAC+, which improves performance by training agents on successful trajectories to reinforce updates leading to higher reward. We evaluate MAC on tagging Personally Identifiable Information (PII), a classification task with limited labels where interpretability is critical, and demonstrate that it generalizes to other agentic tasks such as tool calling. MAC outperforms recent prompt optimization methods by over 50%, produces human-readable and auditable rule sets, and achieves performance comparable to supervised fine-tuning and GRPO without requiring parameter updates.
