Search papers, labs, and topics across Lattice.
This paper introduces a simulation framework that leverages LLM-powered agents conditioned on real user behavioral data to predict A/B test outcomes, significantly reducing the need for actual user traffic and engineering resources. By utilizing data-driven personas derived from anonymized behavioral signals, the approach enhances the fidelity of population modeling compared to traditional synthetic or rule-based methods. The framework achieves a directional accuracy of 0.75-0.90 across 40 A/B tests, highlighting its potential for efficient and cost-effective pre-screening of experiments.
A/B test simulations using data-driven personas can achieve up to 90% accuracy, drastically cutting down on the time and resources needed for real-world testing.
A/B testing is the gold standard for evaluating product changes, but each experiment requires real user traffic, engineering effort, and weeks of measurement. We propose a simulation framework that predicts A/B test outcomes using LLM-powered agents conditioned on data-driven personas grounded in real user behavioral signals. Unlike prior work that relies on synthetic or rule-based personas, our agents are constructed from anonymized behavioral data-activity patterns, engagement signals, and inferred demographics-enabling more faithful population modeling. We frame A/B test simulation as a structured question task and systematically study (i) question design formats, (ii) the impact of persona data source and domain alignment, (iii) the trade-off between per-persona behavioral depth and population diversity, and (iv) efficient population subsampling. On a benchmark of 40 A/B tests spanning two metric types, our best configuration achieves 0.75-0.90 directional accuracy depending on the test metric, demonstrating that data-driven personas are a viable path toward fast, low-cost experiment pre-screening.
Amazon Science