Search papers, labs, and topics across Lattice.
To address dialectal contamination and data loss in regional web corpora, this work establishes an end-to-end curation pipeline that transforms 411 TB of raw Arquivo.pt crawls into a 41-billion-token European Portuguese pre-training dataset. The core technical shift applies boilerplate and line-duplicate removal prior to standard heuristic filtering, preventing premature discards of valid text. This inverted ordering increases final document yield by 19.04% while pairing with dialect-aware language identification, weighted fuzzy deduplication, and neural quality classifiers to deliver a clean, production-grade corpus.
Stripping boilerplate and line duplicates before running heuristic filters rescues 19% of valid web text that standard data pipelines prematurely throw away.
Curating Web corpora for regional language variants like European Portuguese (PT-PT) is heavily bottlenecked by dialectal overlap (mainly with PT-BR) and data processing scale. This paper presents an efficient pipeline to curate a production-ready PT-PT corpus from the Portuguese Web, spanning 411 TB of raw data from Arquivo.pt. We introduce a novel post-scraping block that removes boilerplate and line duplicates prior to filtering. This early-stage intervention increases final document yield by 19.04% by rescuing valid text that standard heuristic filters prematurely discard. Integrated with rigorous language identification, weighted fuzzy deduplication, and neural quality classification, our pipeline offers a scalable framework and a clean, representative corpus optimized for LLM pre-training.
