Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces KARMA, a framework for training LLMs to learn context-sensitive conversational behavior by training a reward model on Reddit conversations to predict response valuation conditioned on context, and then fine-tuning language models via reinforcement learning. Surprisingly, a reward model that was a worse predictor of Reddit karma but relied exclusively on conversational context yielded substantially better downstream performance on pragmatics-mediated tasks. However, KARMA consistently diminished factuality across all conditions, suggesting a tension between pragmatics and factuality embedded in the reward signal.
The best reward model for aligning LLMs with social interaction data isn't the one that best predicts social rewards, but one that focuses on conversational context, revealing a surprising disconnect between reward prediction accuracy and downstream task performance.
Human communication depends on implicit social signals where effectiveness is shaped by tone, context, and conversational norms rather than semantic content alone. We introduce KARMA (Karma-Aligned Reward Model Adaptation), a framework for LLM learning of context-sensitive conversational behavior from large-scale social interaction data. KARMA trains a reward model on Reddit conversations to predict response valuation conditioned on context, and uses this signal to fine-tune language models via reinforcement learning to improve performance on pragmatics-mediated tasks. Critically, we find that the highest performing reward model does not lead to better downstream model alignment: a reward model relying exclusively on conversational context was a worse predictor of Reddit karma but yielded substantially better downstream performance. We evaluate the effects of KARMA applied to a downstream model with and without direct exposure to the social media data. The resulting models show improved pragmatics-mediated behaviors with largely mitigated undesirable side effects. Factuality is consistently diminished by KARMA across all conditions, including when the downstream model has no direct exposure to Reddit data, suggesting that this tension is embedded in the reward signal itself rather than introduced by noisy training data.