Search papers, labs, and topics across Lattice.
This study investigates the efficacy of tool-augmented agentic AI in leveraging data from prior A/B tests to inform the design of new interventions in healthcare messaging. By conducting two-stage field experiments involving over 693,000 patient visits, the researchers compared a Human + Chatbot method with an AI-driven approach that autonomously extracted principles from previous data to generate new messaging variants. The results revealed that the AI-generated interventions significantly outperformed the baseline, with the best message achieving a 69.8% click-through rate, highlighting the importance of domain-specific data over general reasoning capabilities in intervention design.
AI can autonomously generate superior healthcare messaging interventions by learning from past experimental data, outperforming traditional methods by a notable margin.
Organizations routinely run experiments for A/B testing, yet the data generated from one experiment is underutilized to inform subsequent intervention design. Significant barriers exist to extracting actionable knowledge from prior experimental data to inform new interventions. We study whether tool-augmented agentic AI can automatically learn from experimental data to generate new interventions in subsequent experiments. Through two-stage field experiments in healthcare prescription messaging (693,139 patient visits), we compare a Human + Chatbot method (Stage 1: behavioral experts with conversational AI co-designing 13 message variants, 444,691 patient visits) against a Tool-Augmented Agentic AI method (Stage 2: AI autonomously extracting principles from Stage 1 data to generate 17 new variants, 248,448 patient visits). The Agentic AI method, equipped with analytical tools, structured Data-Information-Knowledge-Wisdom (DIKW) reasoning agents, and transparent evidence chains, produces superior interventions: the best AI-generated message achieved a 69.8% CTR (+6.5 percentage points over baseline). Critically, our results suggest that the value comes from domain-specific experimental data, not from general reasoning ability: frontier LLMs operating without experimental data failed to predict which interventions would succeed. The field experiments also revealed that general-purpose behavioral theories used for intervention design do not extend uniformly to specific healthcare contexts, motivating an agentic AI approach to theory audits at field-experiment scale. Our research shows that tool-augmented AI can learn from experimental data and generate improved domain-relevant interventions, transforming behavioral experimentation from one-shot evaluation into a scalable system for cumulative design learning.
