Search papers, labs, and topics across Lattice.
The paper introduces Topic Word Mixing (TWM), a new human evaluation task, to assess inter-topic distinctness in topic models by having annotators distinguish between word sets from single or mixed topics. They compared TWM against word intrusion and automated metrics (coherence, diversity) using six topic models (LDA, NMF, Top2Vec, BERTopic, CFMF, CFMF-emb) on a philosophy of science corpus with nearly 4,000 annotations. Results show that word intrusion and coherence metrics do not always align, especially in specialized domains, while TWM aligns with diversity metrics and captures human-perceived distinctness.
Automated metrics can mislead you: human evaluations of topic models in specialized domains reveal a disconnect between coherence and human-perceived quality, highlighting the need for metrics that better reflect human understanding.
Topic models uncover latent thematic structures in text corpora, yet evaluating their quality remains challenging, particularly in specialized domains. Existing methods often rely on automated metrics like topic coherence and diversity, which may not fully align with human judgment. Human evaluation tasks, such as word intrusion, provide valuable insights but are costly and primarily validated on general-domain corpora. This paper introduces Topic Word Mixing (TWM), a novel human evaluation task assessing inter-topic distinctness by testing whether annotators can distinguish between word sets from single or mixed topics. TWM complements word intrusion's focus on intra-topic coherence and provides a human-grounded counterpart to diversity metrics. We evaluate six topic models - both statistical and embedding-based (LDA, NMF, Top2Vec, BERTopic, CFMF, CFMF-emb) - comparing automated metrics with human evaluation methods based on nearly 4,000 annotations from a domain-specific corpus of philosophy of science publications. Our findings reveal that word intrusion and coherence metrics do not always align, particularly in specialized domains, and that TWM captures human-perceived distinctness while appearing to align with diversity metrics. We release the annotated dataset and task generation code. This work highlights the need for evaluation frameworks bridging automated and human assessments, particularly for domain-specific corpora.
