Search papers, labs, and topics across Lattice.
This paper introduces TriAlign, an innovative offline multi-agent reinforcement learning framework designed to tackle the Truth-Invariant Alignment (TIA) problem in personalized large language models (LLMs). By modeling each social group as an agent, TriAlign optimizes for universal truth accuracy, cross-group truth consistency, and personalization through a fairness-aware objective and an explicit inconsistency penalty. Experimental results show that TriAlign significantly reduces universal truth disparities across social groups while enhancing both objective task performance and personalization quality compared to existing methods.
TriAlign achieves a remarkable balance between universal truth consistency and personalized responses, addressing a critical gap in LLM alignment.
Personalized large language models adapt responses to users' preferences and social attributes, but can introduce substantial universal truth inconsistencies across social groups, where some groups systematically receive less accurate responses on objective tasks. Existing alignment methods either ignore personalization or mainly focus on subjective preference alignment, largely overlooking fairness and consistency in universal truths. To address this gap, we study Truth-Invariant Alignment (TIA), an alignment problem for personalized LLMs that aims to ensure universal truths remain consistent across social groups while preserving personalization. We propose TriAlign, the first offline multi-agent reinforcement learning (MARL) framework for TIA, where each social group is modeled as an agent interacting. TriAlign jointly optimizes universal truth accuracy, cross-group truth consistency, and personalization through a fairness-aware objective and an explicit inconsistency penalty. Experiments across diverse benchmarks demonstrate that TriAlign achieves a stronger balance among these three objectives than strong baselines, reducing universal truth disparities across social groups while improving both objective task performance and personalization quality.
