Search papers, labs, and topics across Lattice.
This paper investigates the challenges in reconstructing constitutional preferences from pairwise preference data used in training language models, highlighting issues of principle quality, composition ambiguity, and variability across models. The authors empirically demonstrate that different executors yield only 73% agreement on principle application, and that constitutions differ significantly between models, with intra-model agreement at 81%. By refining principles through the Inverse Constitutional AI (ICAI+) approach, they achieve improved inter-executor agreement of 78% and comparable accuracy between transparent executors and LLM judges, suggesting a need to evaluate constitutions as integrated systems with executors.
Principle reconstruction in language models is fraught with ambiguity, revealing only 73% agreement between different executors on decision-making.
Pairwise preference data is widely used for training and evaluating language models (e.g., RLHF), but each datapoint records a \emph{choice}, not the rationale behind it. Methods such as Inverse Constitutional AI (ICAI) attempt to improve interpretability by compressing datasets into short ``constitutions'' of natural-language principles. We argue this framing is under-specified: a flat list of principles is not yet an executable decision rule because it leaves principle composition implicit. We use the pairwise setting as a testbed to empirically characterize three open problems in constitutional methods. First, principle quality is hard to measure: coverage and accuracy are useful but incomplete proxies for end-to-end reconstruction. Second, \emph{composition is ambiguous}: holding principles fixed, different executors (LLM judge versus majority vote) agree only $73\%$ of the time. Third, \emph{constitutions differ between LLMs}: cross-model vote agreement is $73\%$, whereas intra-model agreement is $81\%$. Across PRISM, AlpacaEval, and Chatbot Arena, we show that principle refinement (ICAI+) may be a first step towards ameliorating these problems: inter-executor agreement rises to $78\%$, and transparent executors match LLM judge accuracy ($66\%$ vs.\ $67\%$). Our results highlight that constitutions should be evaluated as \emph{constitution--executor systems}, with implications for LLMs-as-a-judge broadly.
