Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
To address the behavioral collapse of flat role-playing prompts in agent evaluations, this work operationalizes a 23-dimensional user persona vector spanning categorical demographics, perturbed continuous behavioral traits, and scenario-reactive emotional states paired with a query-complexity overlay. Evaluated across 64,698 multi-turn conversations across three production corpora, the framework drives a 15.8 percentage-point spread in agent goal completion and up to a 20 percentage-point gap in multi-turn workflow compliance. This yields an auditable, highly controllable benchmark methodology that faithfully reproduces real-world user difficulty distributions without requiring learned covariance matrices.
Flat user-simulation prompts dangerously mask agent vulnerabilities: decomposing users into 23 demographic, behavioral, and dynamic emotional dimensions exposes a 15.8 percentage-point variance in agent task success under realistic human friction.
Evaluating tool-augmented LLM agents requires diverse, realistic user inputs yet most evaluation frameworks use flat role descriptions ("you are an angry customer") that produce near-identical conversations regardless of the underlying scenario. In this paper, we propose a three-tier persona vector with 23 operationalized dimensions: 6 categorical demographics (jurisdiction, age, channel, device, language proficiency, time availability), 12 continuous behavioral traits (patience, assertiveness, digital literacy, etc.) sampled with Gaussian noise around curated profile base vectors, and 5 continuous emotional states (frustration, anxiety, trust, confidence, stress) that shift in response to scenario context. Orthogonal to the persona, a 4-level query-complexity overlay controls utterance phrasing from direct to deliberately vague. We evaluate the persona model inside a synthetic data generation pipeline across 64,698 multi-turn conversations spanning 8 named profiles and 3 production corpora. Key findings: (i) a 15.8 percentage-point spread in agent goal-achievement across personas confirms trait vectors produce measurably different user behavior; (ii) the same persona behaves differently across scenarios due to scenario-reactive emotional state shifts, validating the scenario-reactive design; (iii) domain-specific projects show persona sensitivity on booking-flow compliance (~15-20 percentage points gap between tier-aware and pressure-test personas), demonstrating the model faithfully reproduces real-world difficulty distributions; (iv) seven rule-described trait correlations produce auditable co-occurrence patterns without requiring learned covariance matrices. The persona model is fully specified for reproduction.