[Paper Review] Learning to Denoise Distantly-Labeled Data for Entity Typing

Yasumasa Onoe, Greg Durrett|arXiv (Cornell University)|May 4, 2019

Topic Modeling31 references4 citations

TL;DR

This paper proposes a two-stage denoising framework for distantly labeled data in ultra-fine entity typing, using a learned filtering function to discard unusable examples and a relabeling function to correct noisy labels. Trained on synthetically noised data, the method improves performance by 3 F1 over an ELMo-based baseline and achieves state-of-the-art results, outperforming both raw distant data and heuristic denoising approaches.

ABSTRACT

Distantly-labeled data can be used to scale up training of statistical models, but it is typically noisy and that noise can vary with the distant labeling technique. In this work, we propose a two-stage procedure for handling this type of data: denoise it with a learned model, then train our final model on clean and denoised distant data with standard supervised training. Our denoising approach consists of two parts. First, a filtering function discards examples from the distantly labeled data that are wholly unusable. Second, a relabeling function repairs noisy labels for the retained examples. Each of these components is a model trained on synthetically-noised examples generated from a small manually-labeled set. We investigate this approach on the ultra-fine entity typing task of Choi et al. (2018). Our baseline model is an extension of their model with pre-trained ELMo representations, which already achieves state-of-the-art performance. Adding distant data that has been denoised with our learned models gives further performance gains over this base model, outperforming models trained on raw distant data or heuristically-denoised distant data.

Motivation & Objective

• To address the challenge of noisy labels in distantly supervised entity typing, where distant supervision sources like entity linking and head words introduce inconsistent or incomplete labels.
• To improve model performance by transforming noisy distant data into high-quality training examples without modifying the final model's training pipeline.
• To develop a differentiable, end-to-end denoising approach that conditions on both input and noisy labels, enabling label space structure learning.
• To evaluate the method on the ultra-fine entity typing benchmark of Choi et al. (2018), which features over 10,000 label types across three granularity levels.
• To demonstrate that learned denoising generalizes beyond the original label set, improving performance even when labels are projected onto the OntoNotes schema.

Proposed method

• A filtering function is trained to identify and discard distantly labeled examples deemed too noisy or unreliable, based on synthetic noise injected into a small manually-labeled dataset.
• A relabeling function is trained to transform noisy labels into more accurate, human-annotator-like labels by leveraging joint embeddings of entities and labels.
• Both functions are trained using a conditional denoising autoencoder-like objective on synthetically noised examples, enabling the model to learn corrections from noisy-to-true label mappings.
• The denoising models are designed to mirror the architecture of the final entity typing model, using ELMo embeddings for both input and label representations.
• The final model is trained on a combination of manually labeled data and denoised distant data, using standard supervised learning with no changes to the training procedure.
• The approach is evaluated on two distant supervision sources: entity linking (EL) and head word heuristics (HEAD), using the ultra-fine entity typing dataset from Choi et al. (2018).

Experimental results

Research questions

• RQ1 Can a learned denoising model improve performance on ultra-fine entity typing when applied to distantly labeled data?
• RQ2 How does the performance of a model trained on denoised distant data compare to models trained on raw distant data or heuristically denoised data?
• RQ3 To what extent does the denoising process recover missing or incorrect labels, particularly ultra-fine types?
• RQ4 Does the denoising framework generalize to different label schemas, such as the OntoNotes schema?
• RQ5 Can the filtering and relabeling functions effectively identify and correct label noise while preserving useful examples?

Key findings

• The proposed denoising framework achieves a 3.0 F1 absolute improvement over the ELMo-based baseline model, establishing a new state-of-the-art on the ultra-fine entity typing benchmark.
• The method improves performance by 4.4 F1 over naive training on raw distant data, demonstrating that denoising is essential for leveraging distant supervision effectively.
• The relabeling function adds an average of 2.03 ultra-fine types per EL example and 1.15 per HEAD example, significantly recovering missing fine-grained and ultra-fine labels.
• The filtering function discards 9.4% of EL examples and 10.0% of HEAD examples, indicating a substantial portion of distant data is too noisy to be useful.
• The denoised data improves performance even when labels are projected onto the OntoNotes schema, outperforming the original method of Choi et al. (2018) in that setting.
• The results show that end-to-end learned denoising, trained on synthetic noise, generalizes better than heuristic-based filtering and relabeling for this task.