Search papers, labs, and topics across Lattice.
This paper introduces Granuscore, a novel reference-free metric for quantifying granularity in text by analyzing structural properties within a hierarchical embedding space. The authors demonstrate Granuscore's ability to recover hierarchical orderings on the Granola-EQ dataset and capture granularity variations across discourse contexts, while also showing its correlation with sentence specificity beyond length. They further apply Granuscore to question-answering benchmarks, revealing granularity differences between questions, gold answers, and model outputs, offering insights into model behavior and dataset difficulty.
Finally, a way to measure the "zoom level" of text without needing a reference, opening up new possibilities for understanding model behavior and dataset difficulty in QA.
Natural language conveys information at varying levels of granularity, from fine-grained references to broad descriptions. While granularity is fundamental to human communication, existing measures mostly capture surface detail or sentence specificity. We introduce Granuscore, a reference-free measure of granularity that leverages structural properties of a hierarchical embedding space. Granuscore reliably recovers hierarchical orderings on the Granola-EQ dataset and captures expected differences in granularity across discourse contexts. Across domains, we further show that Granuscore explains non-linear variation in sentence specificity beyond sentence length. Finally, we apply Granuscore to four question-answering benchmarks and analyze how granularity differs for questions, gold answers, and model outputs across response outcomes. The analysis reveals consistent differences in model behavior and provides a principled lens for characterizing the difficulty of QA datasets. Together, the results position Granuscore as a scalable, broadly applicable tool for analyzing granularity in text.
