Search papers, labs, and topics across Lattice.
The Institutional Newspapers Pipeline was developed to extract structured datasets from historical newspaper scans, addressing the challenges posed by their irregular layouts. By utilizing a multi-step process that includes segmentation, OCR, and various forms of text analysis, the pipeline successfully generated 16.3 billion tokens from over 1.4 million public domain newspaper scans. This work not only provides a valuable open dataset but also establishes a customizable and interpretable framework for future research on historical texts.
Unlocking 16.3 billion tokens from historical newspapers could revolutionize access to archival data for AI research and applications.
Historical newspapers are an abundant record of public life, but their dense, irregular and sometimes noisy layouts make computational access to these materials both challenging and limited. We present the Institutional Newspapers Pipeline, a modular system we jointly designed with Boston Public Library to extract high-quality, structured datasets from historical newspaper scans. It was architected so that each step remains interpretable and customizable, and so that the pipeline as a whole remains computationally frugal enough to run on workstation-level hardware. The pipeline runs each scan through a multi-step process: it segments scans into individual type-agnostic crops and performs OCR on each resulting segment before then performing text analysis, type classification, reading order detection, named entities recognition, subject classification, language detection, and pre-computed embeddings generation on every crop. We ran this pipeline against a portion of Boston Public Library's holdings and released the results as an open dataset. The optical character recognition (OCR) output represents 16.3 billion o200k_base tokens across 83.1 million individual crops, extracted from 1,473,635 public domain newspaper scans published between 1795 and 1930. This report describes our methods for each processing step, the small models we trained, as well as the evaluation results and dataset-scale measurements we collected in the process. It accompanies the release of the pipeline, models, and dataset. We position this work as a substantial step towards unlocking high-quality data from tens of millions of newspaper scans.
