Search papers, labs, and topics across Lattice.
This paper introduces PolyAlign, a novel framework for conditional human-distribution alignment that addresses the limitations of traditional post-training methods like supervised fine-tuning (SFT) and preference optimization, which often enforce a single global assistant behavior. By organizing bilingual interaction data into context-specific human reference distributions, PolyAlign employs Bucket-Aware SFT and Human-Distribution Preference Optimization (HDPO) to enhance the natural variation in model responses across different languages and interaction contexts. The framework demonstrates significant improvements in conditional naturalness and distributional faithfulness in bilingual evaluations, suggesting a shift towards more context-aware alignment strategies in language models.
PolyAlign reveals that aligning models to context-specific human response distributions significantly enhances naturalness and faithfulness in multilingual interactions.
Post-training methods such as supervised fine-tuning (SFT) and preference optimization typically align language models toward a single global assistant behavior. While effective for improving average helpfulness, this can suppress the natural variation of human responses across languages, tasks, and dialogue settings. We study this problem as conditional human-distribution alignment: models should match the human response distribution appropriate to the current interaction context, rather than a universal response style. We introduce PolyAlign, a distribution-aware alignment framework that organizes bilingual interaction data into bucket-specific human reference distributions defined by language, interaction track, response family, and length. PolyAlign combines Bucket-Aware SFT, which balances optimization across heterogeneous buckets, with Human-Distribution Preference Optimization (HDPO), which regularizes preference learning using critic-estimated distance to bucket-specific human support. Across a bilingual evaluation suite covering English and Chinese single- and multi-turn settings, PolyAlign improves conditional naturalness and distributional faithfulness while preserving competitive task utility. The results suggest that post-training should move beyond global alignment objectives toward interaction-aware alignment with human response distributions.
