Search papers, labs, and topics across Lattice.
The paper introduces A/B Agent, a self-evolving framework designed to optimize industrial recommendation strategies through a closed-loop A/B testing process. By organizing historical strategies into a hierarchical experience tree and employing a multi-path Tree-RAG for strategy generation, the agent autonomously refines its recommendations based on real-time A/B feedback. The results show a significant 4.829% increase in gross merchandise value (GMV) in a real-world application, demonstrating the agent's effectiveness in enhancing performance while adhering to safety metrics.
A/B Agent achieves a 4.829% boost in GMV by autonomously evolving recommendation strategies through a novel hierarchical knowledge organization and real-time feedback loop.
Industrial recommendation strategy iteration heavily relies on large-scale A/B experimentation. Traditional tuning requires experts to repeatedly design strategies, configure experiments, analyze results, and adjust parameters, making the process labor-intensive and time-consuming. Meanwhile, valuable knowledge from historical experiments is often fragmented, making systematic reuse difficult through manual expert effort alone. Existing RAG agents partially alleviate this burden by retrieving prior strategies, but typically organize experience in a flat manner, overlooking the hierarchical relationships among business scenarios, recommendation stages, optimization objectives, and experimental contexts. This often results in mismatched retrieval and limited cross-scenario transfer, while preventing agents from continuously refining strategies and parameters through sequential A/B feedback. % To address these limitations, we propose A/B Agent, a closed-loop A/B agent for industrial recommendation strategy optimization. The framework comprises three tightly coupled core components: Historical Strategy Knowledge Organization, Autonomous Target-Aware Strategy Generation, and Experiment-Guided Strategy Self-Evolution. It organizes historical strategies into a hierarchical experience tree, retrieves transferable evidence through multi-path Tree-RAG to generate executable strategies, and continuously analyzes online A/B feedback to guide autonomous tuning and update the experience tree for self-evolution. Extensive offline and online evaluations demonstrate its effectiveness, including a 4.829% improvement in GMV in a real-world short-video e-commerce recommendation system while maintaining positive gains across all guardrail metrics.
