Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces news-crawler-LM, a compact long-context language model specifically fine-tuned for extracting structured content from diverse news page formats. By leveraging high-quality, human-validated extractions, the model effectively converts raw HTML into plaintext and structured JSON, significantly outperforming existing baselines in both HTML-to-Markdown and HTML-to-JSON tasks. Notably, while it excels in structured extraction, its performance on HTML-to-plaintext tasks is only marginally better than traditional rule-based parsers when applied to new publishers.
A small, fine-tuned language model can outperform traditional baselines in extracting structured news content, streamlining the crawling process without extensive manual configuration.
Extracting structured content from news pages remains challenging due to heterogeneous HTML layouts, inconsistent markup, and substantial boilerplate such as navigation elements and advertisements. Rule-based news crawlers can achieve high extraction accuracy by encoding site-specific structure, but require manual configuration in order to generalize to new publishers. Large language models provide a more flexible alternative by reducing the need for handcrafted rules, but their high computational cost limits practical deployment. In this paper, we introduce news-crawler-LM, a small long-context language model fine-tuned on high-quality, human-validated extractions from the Fundus news-crawling library. Our model converts raw HTML into plaintext and structured JSON, including fields such as headline, author, publication date, and article body. In our experiments, news-crawler-LM outperforms strong baselines in HTML-to-Markdown and HTML-to-JSON extraction, improving performance by +4.8 BLEU and +6.1 METEOR in the HTML-to-Markdown task, and by +2.2 BLEU and +4.1 METEOR in the HTML-to-JSON task. However, we also observe that our model only slightly better compared to other rule-based parsing libraries on the HTML-to-plaintext task in evaluations on previously unseen publishers. We release all models and artifacts to the research community.