Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper critically evaluates SemEval-2020 Task 1, a key benchmark for lexical semantic change detection, across three dimensions: operationalization, data quality, and benchmark design. It argues that the task's framing of semantic change as discrete sense shifts is overly simplistic and that the benchmark suffers from significant corpus and preprocessing errors. Furthermore, the paper identifies limitations in the benchmark design, including small target sets and limited language coverage, which reduce realism and increase statistical uncertainty.
SemEval-2020 Task 1, the most influential benchmark for lexical semantic change detection, may be more of a noisy proxy than a gold standard due to flawed operationalization, poor data quality, and unrealistic design.
This discussion paper re-examines SemEval-2020 Task 1, the most influential shared benchmark for lexical semantic change detection, through a three-part evaluative framework: operationalisation, data quality, and benchmark design. First, at the level of operationalisation, we argue that the benchmark models semantic change mainly as gain, loss, or redistribution of discrete senses. While practical for annotation and evaluation, this framing is too narrow to capture gradual, constructional, collocational, and discourse-level change. Also, the gold labels are outcomes of annotation decisions, clustering procedures, and threshold settings, which could potentially limit the validity of the task. Second, at the level of data quality, we show that the benchmark is affected by substantial corpus and preprocessing problems, including OCR noise, malformed characters, truncated sentences, inconsistent lemmatisation, POS-tagging errors, and missed targets. These issues can distort model behaviour, complicate linguistic analysis, and reduce reproducibility. Third, at the level of bench-mark design, we argue the small curated target sets and limited language coverage reduce realism and increase statistical uncertainty. Taken together, these limitations suggest that the benchmark should be treated as a useful but partial test bed rather than a definitive measure of progress. We therefore call for future datasets and shared tasks to adopt broader theories of semantic change, document pre-processing transparently, expand cross-linguistic coverage, and use more realistic evaluation settings. Such steps are necessary for more valid, interpretable, and generalisable progress in lexical semantic change detection