Search papers, labs, and topics across Lattice.
The paper introduces WebFAQ 2.0, a large-scale multilingual question-answering dataset comprising 198 million FAQ pairs across 108 languages, significantly expanding upon the original WebFAQ dataset with a novel web crawling and extraction strategy. To address the need for improved training of dense retrievers, the authors release a hard negatives dataset containing 1.25M queries across 20 languages, mined using a two-stage retrieval pipeline and scored with cross-encoders. Experiments demonstrate the utility of WebFAQ 2.0 and its hard negatives for fine-tuning dense retrievers using both Contrastive Learning and Knowledge Distillation approaches.
Multilingual QA just got a massive boost: WebFAQ 2.0 offers 198M FAQ pairs across 108 languages, plus mined hard negatives to sharpen your dense retrievers.
We introduce WebFAQ 2.0, a new version of the WebFAQ dataset, containing 198 million FAQ-based natural question-answer pairs across 108 languages. Compared to the previous version, it significantly expands multilingual coverage and the number of bilingual aligned QA pairs to over 14.3M, making it the largest FAQ-based resource. Unlike the original release, WebFAQ 2.0 uses a novel data collection strategy that directly crawls and extracts relevant web content, resulting in a substantially more diverse and multilingual dataset with richer context through page titles and descriptions. In response to community feedback, we also release a hard negatives dataset for training dense retrievers, with 1.25M queries across 20 languages. These hard negatives were mined using a two-stage retrieval pipeline and include cross-encoder scores for 200 negatives per query. We further show how this resource enables two primary fine-tuning strategies for dense retrievers: Contrastive Learning with MultipleNegativesRanking loss, and Knowledge Distillation with MarginMSE loss. WebFAQ 2.0 is not a static resource but part of a long-term effort. Since late 2025, structured FAQs are being regularly released through the Open Web Index, enabling continuous expansion and refinement. We publish the datasets and training scripts to facilitate further research in multilingual and cross-lingual IR. The dataset itself and all related resources are publicly available on GitHub and HuggingFace.
