Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the issue of annotation noise in large language model (LLM) outputs used for training named entity recognition (NER) systems, revealing that existing fine-tuning methods overlook the heterogeneous nature of this noise. The authors propose an error-type-aware loss reweighting approach that applies distinct reweighing rules for different types of errors, enhancing the model's ability to learn from noisy labels. Their method demonstrates a significant performance improvement, achieving F1 score gains of up to 4.6 percentage points under specific noise conditions on the Wikigold dataset.
Ignoring the nuances of annotation noise can lead to substantial performance degradation in NER models, but tailored loss reweighting can recover critical training signals.
Large language models are increasingly used to annotate datasets for training smaller, task-specialized models such as named entity recognition. While this method yields effective models, it assumes that the synthetic dataset is correctly annotated. In this work, we find that (i) current fine-tuning processes simply ignore LLM-introduced annotation noise, resulting in degraded performance and (ii) existing noise-robust losses are not transferable to sequence labeling because annotation noise in named entity recognition is heterogeneous: for example, missing mentions and type errors affect the training signal in different ways. Treating all noisy tokens equally in noise-robust losses and applying a single reweighing criterion for all may therefore remove useful supervision or reinforce incorrect labels. To address this limitation, we propose error-type-aware loss reweighting for NER, which introduces separate reweighing rules for different types of potentially erroneous tokens. Our approach is simple and efficient, does not require additional training resources, and improves F1 by 0.8 - 2.0 percentage points on dataset-level average for noise levels between 15% and 40%, with a maximum improvement of 4.6 percentage points with 24.1% noise on Wikigold.