Search papers, labs, and topics across Lattice.
The Institutional Books - Enriched Text (IB-HL-ET) project introduces a customizable open-source pipeline designed to enhance the processing of a vast collection of 983,004 volumes from Harvard Library. By focusing on preserving metadata and providing detailed annotations, the pipeline allows users to tailor text outputs according to specific needs, rather than relying on a one-size-fits-all approach that often discards valuable information. Key results include the generation of 217B tokens organized into 1.39B annotated subtopic paragraphs, facilitating both machine parsing and human study across approximately 250 languages.
Tailor your text analysis with a revolutionary pipeline that preserves metadata while enhancing OCR text processing for 983,004 volumes.
Released in 2025, Institutional Books: Harvard Library (IB-HL) is a collection of 983,004 volumes (242B o200k_base tokens), originally digitized through Harvard Library's participation in the Google Books Library project. As researchers and developers have begun to use IB-HL, a tension has emerged between standard large-scale preprocessing practices and the goals of careful information stewardship. Many existing pipelines optimize for web text: as a result, they tend to aggressively filter, deduplicate, restrict by language, and sometimes discard meaningful metadata. Meanwhile, researchers seeking to use IB-HL duplicate effort while performing similar processing and analysis. We describe an approach that we call Enriched Text. Instead of producing a single'complete'stream of tokens, we normalize the text while preserving metadata through annotations. We separate endmatter, detect per-paragraph language, identify clusters of duplicate paragraphs, and compute per-paragraph bits-per-byte scores. We provide this information through HTML-like annotations layered on top of the text. By parsing these annotations, users can tailor the output to their own needs instead of accepting a global editorial decision on content. The pipeline applies to all $\approx$250 languages in the collection. This report describes this project's goals, implementation, and design rationale. The release includes IB-HL-ET (an enriched-text version of IB-HL containing 217B o200k_base tokens across 983,003 volumes, organized into 1.39B annotated subtopic paragraphs) and the pipeline that produced it. These serve to make the collection easier for machines to parse and for humans to study.
