Search papers, labs, and topics across Lattice.
The paper analyzes coreference-based evaluations of NLU, revealing issues with measurement validity, specifically contestedness and convergent validity, that limit the generalizability of conclusions. It introduces a novel evaluation method focused on inferring the relative plausibility of events to assess coreference resolution. The evaluation demonstrates that while language models show improved performance on standard benchmarks, they exhibit sensitivity to evaluation conditions and struggle to generalize as humans would.
Coreference benchmarks may be overstating language models' NLU abilities, as even small changes to evaluation contexts reveal a failure to generalize.
In this thesis, I refine our understanding as to what conclusions we can reach from coreference-based evaluations by expanding existing evaluation practices and considering the extent to which evaluation results are either converging or conflicting. First, I analyze standard coreference evaluations and show that their design often leads to non-generalizable conclusions due to issues of measurement validity - including contestedness (multiple, competing definitions of coreference) and convergent validity (evaluation results that rank models differently across benchmarks). Second, I propose and implement a novel evaluation focused on testing systems'ability to infer the relative plausibility of events, a key aspect of resolving coreference. Through this extended evaluation, I find that contemporary language models demonstrate strong performance on standard benchmarks - improving over earlier baseline systems within certain domains and types of coreference - but remain sensitive to the evaluation conditions: they often fail to generalize in ways one would expect a human to be capable of when evaluation contexts are slightly modified. Taken together, these findings clarify both the strengths, such as improved accuracy over baselines on widely used evaluations, and the limitations of the current NLP paradigm, including weaknesses in measurement validity, and suggest directions for future work in developing better evaluation methods and more genuinely generalizable systems.
Mila