Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces a reinforcement learning-based web crawling algorithm, SB-CLASSIFIER, designed to efficiently acquire statistical datasets (SDs) from websites. The algorithm addresses the challenge of inefficient or impossible SD retrieval at scale by learning which hyperlinks lead to pages that link to many targets, based on the paths leading to the links in their enclosing webpages. Experiments on large websites demonstrate that SB-CLASSIFIER can retrieve a high fraction of a site's targets while crawling only a small part of the website.
Reinforcement learning can dramatically improve web crawling efficiency, allowing for high-quality statistical datasets to be acquired while exploring only a fraction of a website.
Journalistic fact-checking, as well as social or economic research, require analyzing high-quality statistics datasets (SDs, in short). However, retrieving SD corpora at scale may be hard, inefficient, or impossible, depending on how they are published online. To improve open statistics data accessibility, we present a focused Web crawling algorithm that retrieves as many targets, i.e., resources of certain types, as possible, from a given website, in an efficient and scalable way, by crawling (much) less than the full website. We show that optimally solving this problem is intractable, and propose an approach based on reinforcement learning, namely using sleeping bandits. We propose SB-CLASSIFIER, a crawler that efficiently learns which hyperlinks lead to pages that link to many targets, based on the paths leading to the links in their enclosing webpages. Our experiments on websites with millions of webpages show that our crawler is highly efficient, delivering high fractions of a site's targets while crawling only a small part.