Track new results across pure and applied mathematics.
Search papers, labs, and topics across Lattice.
GroupTravelBench is introduced as the first benchmark for multi-user, multi-turn travel planning, designed to evaluate LLM agents' abilities in elicitation, coordination, and planning. The benchmark comprises 650 synthesized tasks based on real user profiles, POI data, and ticket prices, spanning three difficulty levels. Evaluation of various LLMs on GroupTravelBench reveals significant shortcomings in preference coverage and group fairness, highlighting areas for improvement in LLM agent design.
Frontier LLMs still struggle with preference coverage and group fairness when planning travel for multiple users, revealing a critical gap in real-world agent capabilities.
Travel planning is a realistic task for evaluating the planning and tool-use abilities of LLM agents. However, existing benchmarks typically assume only a single user, thereby avoiding one of the most challenging aspects of real-world scenarios: an agent's ability to identify and resolve conflicts among multiple users. To address this gap, we introduce \textbf{GroupTravelBench}, the first benchmark for \textbf{multi-user, multi-turn} travel planning. Based on real user profiles, POI data, and ticket price data, we synthesize 650 tasks and divide them into three difficulty levels. Beyond standard abilities in single-user itinerary planning, such as multi-step reasoning and tool use, our benchmark further evaluates three key capabilities required for travel agents: \emph{(i) elicitation} -- proactively engaging in multi-turn dialogue to gather preferences from each user; \emph{(ii) coordination} -- resolving conflicts among users through compromise or subgrouping strategies; and \emph{(iii) planning} -- searching for travel plans that maximize overall group utility while maintaining fairness and feasibility. To simulate real-world conversational itinerary planning while enabling reliable tool use and offline evaluation, we build an interactive sandbox environment with cached real-world tool data. We evaluate a wide range of LLMs and find that even frontier models still show substantial weaknesses in preference coverage and group fairness. \textit{GroupTravelBench} provides a practical and reproducible benchmark for advancing research on LLM agents for real-world travel planning.
DAMO