Search papers, labs, and topics across Lattice.
This paper introduces SynthAVE, a scalable framework for synthetic labeling in e-commerce attribute extraction, addressing the challenge of obtaining large, diverse labeled datasets across multiple languages and product types. By employing a multi-LLM arena framework, the authors validate synthetic labels through a majority voting system involving 21 judge configurations, achieving a high agreement rate with human experts. The results indicate that leveraging diverse models can yield reliable predictions, significantly reducing the cost of data labeling while ensuring quality remains on par with human annotations.
Majority voting among diverse LLMs achieves 95.2% agreement with human experts, making synthetic labeling for e-commerce both scalable and cost-effective.
Fine-tuning large language models (LLMs) for e-commerce attribute extraction requires labeled data representative across thousands of product types, attributes, and multiple languages. This combinatorial scale translates to millions of annotations, rendering human labeling prohibitively costly. While recent work has demonstrated synthetic label generation using LLMs, deploying such approaches at industrial scale requires integrated quality control mechanisms. We present SynthAVE, a large-scale human-validated benchmark for attribute value extraction spanning 12,726 products across 229 product types, 792 attributes, and 4 languages (Spanish, French, Italian, German). To validate synthetic labels at scale, we introduce a multi-LLM arena framework where samples are independently evaluated by 21 judge configurations (7 model families $\times$ 3 prompts), with final labels determined via majority voting. The majority vote ensemble agrees with human experts at Cohen's $\kappa = 0.92$ (95.2% agreement), while individual judges show substantial inter-model agreement (Fleiss'$\kappa = 0.76$). This demonstrates that diverse models with varying individual judgments aggregate into highly reliable predictions, enabling cost-effective validation at scale while maintaining quality parity with human review.
