Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study replicates prior findings on human label variation in natural language inference by examining unselected populations from SNLI and MultiNLI, contrasting with earlier work that focused on selected items. Contrary to the original prediction that non-upward monotonicity operators would show lower label agreement, the results reveal a slight increase in agreement among these items. The findings suggest that previous conclusions regarding label variation may be artifacts of the selection process rather than inherent properties of the data, emphasizing the need for transparency in selection criteria for NLI studies.
Label agreement in natural language inference may be higher than previously thought, challenging assumptions based on selected datasets. WHY_IT MATTERS: This insight could fundamentally alter how researchers interpret human label variation and the reliability of conclusions drawn from selected versus unselected data.
Prior work on human label variation (HLV) in natural language inference (NLI) has often relied on re-annotation resources that select items by disagreement level. An earlier study (arXiv:2607.15870) found that hypotheses containing non-upward monotonicity operators showed lower label agreement in ChaosNLI (Cliff's delta = -0.284), which is restricted to items whose majority label carries exactly three of five votes. We preregistered a replication of this boundary in the unselected populations that ChaosNLI was drawn from: the SNLI and MultiNLI development sets, using the same operator tagger and a four-level ordinal agreement outcome. The registered prediction fails. All seven contrasts return a positive Cliff's delta (non-upward items agree slightly more, not less), the only significant confirmatory contrast has the opposite sign to the registration, and every effect is far below our smallest effect size of interest (0.10). Robustness checks support the measurement: simulated tagger misclassification shrinks the effects rather than manufacturing them, and a manual re-tagging audit reaches four-class agreement of 0.875 on a fresh 200-item sample. We conclude that the earlier negative boundary is plausibly a structure conditional on low-agreement selection rather than a population-level property, and that HLV structure claims built on selected re-annotation resources should state their selection conditional explicitly.