Search papers, labs, and topics across Lattice.
The authors introduce ScrapeGraphAI-100k, a large-scale dataset of 93,695 real-world web information extraction events collected via opt-in telemetry from the ScrapeGraphAI platform. This dataset addresses the limitations of existing web extraction datasets by providing structural context, diverse domains, and multiple languages. Fine-tuning a 1.7B parameter model on a subset of the dataset demonstrates its utility by significantly narrowing the performance gap with larger 30B parameter baselines.
Training a small 1.7B parameter model on the new ScrapeGraphAI-100k dataset lets it nearly match the web information extraction performance of models almost 20x larger.
The use of large language models for web information extraction is becoming increasingly fundamental to modern web information retrieval pipelines. However, existing datasets tend to be small, synthetic or text-only, failing to capture the structural context of the web. We introduce ScrapeGraphAI-100k, a large-scale dataset comprising real-world LLM extraction events, collected via opt-in ScrapeGraphAI telemetry during Q2 and Q3 of 2025. Starting from 9M events, we deduplicate and balance by schema to produce 93,695 examples spanning diverse domains and languages. Each instance includes Markdown content, a prompt, a JSON schema, the LLM response, and complexity/validation metadata. We characterize the datasets structural diversity and its failure modes as schema complexity increases. We also provide a fine-tuning experiment showing that a small language model (1.7B) trained on a subset narrows the gap to larger baselines (30B), underscoring the datasets utility for efficient extraction. ScrapeGraphAI-100k enables fine-tuning small models, benchmarking structured extraction, and studying schema induction for web IR indexing, and is publicly available on HuggingFace.
