Search papers, labs, and topics across Lattice.
This paper introduces SoCRATES, a novel benchmark designed for the evaluation of proactive LLM mediators across diverse, realistic conflict scenarios. By utilizing an agentic pipeline to construct scenarios from real conflicts and employing a topic-localized evaluator, the authors achieve a remarkable alignment score of 0.82 with human experts, significantly improving upon previous evaluation methods. The findings reveal that even the most advanced mediators only manage to close about one-third of the consensus gap, emphasizing the critical need for social adaptation in mediation processes.
Even the best LLM mediators struggle to bridge the consensus gap in realistic conflict scenarios, closing only a third of it under diverse socio-cognitive conditions.
Evaluating LLM mediators remains challenging, as mediation unfolds as a real-time trajectory shaped by disputants'shifting emotions, intentions, and context. Existing testbeds rely on a few expert-authored domains, vary mainly strategic posture, and score every turn against every topic, introducing off-topic noise. We introduce SoCRATES, a benchmark for evaluating proactive LLM mediators in realistic, multi-domain testbeds. It constructs scenarios from real conflicts through an agentic pipeline across eight domains, probes five socio-cognitive adaptation axes (strategic posture, party composition, history length, emotional reactivity, and cultural identity), and scores each topic only on the turns that advance it via a topic-localized evaluator. The evaluator reaches 0.82 alignment with human experts, more than doubling a per-turn baseline. Benchmarking eight frontier LLMs, we find that even the strongest mediator closes only about a third of the unmediated consensus gap under diverse and realistic testbeds, with performance varying sharply by socio-cognitive axis, highlighting that progress lies in social adaptation to diverse conditions.
