Search papers, labs, and topics across Lattice.
This paper benchmarks seven document selection strategies, ranging from random to hybrid retrieval, to assess their impact on downstream text analysis tasks using LDA, BERTopic, TopicGPT, and HiCode. They find that semantic or hybrid retrieval methods consistently outperform simpler strategies without incurring excessive computational costs. The study highlights document selection as a critical methodological choice that significantly influences the results of query-focused text analysis.
Seemingly innocuous choices about document selection can drastically alter the outcomes of text analysis, but semantic or hybrid retrieval offers a robust, efficient solution.
Analyses of document collections often require selecting what data to analyze, as not all documents are relevant to a particular research question and computational constraints preclude analyzing all documents, yet little work has examined effects of selection strategy choices. We systematically evaluate seven selection methods (from random selection to hybrid retrieval) on outputs from four text analyses methods (LDA, BERTopic, TopicGPT, HiCode) over two datasets with 26 open-ended queries. Our evaluation reveals practice guidance: semantic or hybrid retrieval offer strong go-to approaches that avoid the pitfalls of weaker selection strategies and the unnecessary compute overhead of more complicated ones. Overall, our evaluation framework establishes data selection as a methodological decision, rather than a practical necessity, inviting the development of new strategies.
