[Paper Review] Robust Distant Supervision Relation Extraction via Deep Reinforcement Learning

Pengda Qin, Weiran Xu+1 more|arXiv (Cornell University)|May 24, 2018

Topic ModelingComputer Science20 references45 citations

TL;DR

The paper introduces a policy-based deep reinforcement learning framework to identify and redistribute false-positive samples in distantly supervised relation extraction, improving robustness across models without changing the underlying relation classifiers.

ABSTRACT

Distant supervision has become the standard method for relation extraction. However, even though it is an efficient method, it does not come at no cost---The resulted distantly-supervised training samples are often very noisy. To combat the noise, most of the recent state-of-the-art approaches focus on selecting one-best sentence or calculating soft attention weights over the set of the sentences of one specific entity pair. However, these methods are suboptimal, and the false positive problem is still a key stumbling bottleneck for the performance. We argue that those incorrectly-labeled candidate sentences must be treated with a hard decision, rather than being dealt with soft attention weights. To do this, our paper describes a radical solution---We explore a deep reinforcement learning strategy to generate the false-positive indicator, where we automatically recognize false positives for each relation type without any supervised information. Unlike the removal operation in the previous studies, we redistribute them into the negative examples. The experimental results show that the proposed strategy significantly improves the performance of distant supervision comparing to state-of-the-art systems.

Motivation & Objective

• Motivate and address noise in distantly supervised relation extraction.
• Develop a model-independent RL-based method to identify false positives without manual labeling.
• Demonstrate that redistributing false positives improves performance of existing neural relation extractors.
• Showcase robustness across multiple baselines on the NYT-Freebase dataset.

Proposed method

• Formulate distant supervision as an MDP by integrating state info from current and prior sentences.
• Use a policy network (CNN-based) to decide remove vs retain for each distantly supervised sentence per relation type.
• Pre-train the policy network with a supervised-like step using a highly imbalanced DS positive/negative split.
• Train the RL agent to remove a fixed number of sentences per epoch and redistribute them to negative set, using reward based on improvement in F1 on a validation set.
• Define reward R_i as alpha times the difference in F1 between consecutive epochs, averaged over the last five epochs to stabilize training.
• Redistribute the removed samples to negative sets and retrain the relation classifier to evaluate reward-driven performance.

Experimental results

Research questions

• RQ1 Can a policy-based RL agent learn to reliably identify false-positive sentences in distantly supervised data without manual labeling?
• RQ2 Does redistributing false positives to the negative set improve the performance of existing relation extraction models on NYT-Freebase?
• RQ3 Is the proposed RL framework model-independent and compatible with different neural relation extractors?
• RQ4 What is the impact of pre-training and reward-driven retraining on classifier performance?

Key findings

• The RL-based false-positive indicator improves F1 scores of relation classifiers compared to using original or pre-trained strategies alone.
• Pre-training the policy network yields noticeable improvements, and RL retraining provides further gains for several relation types.
• RL-enhanced training yields improved area under PR curves (AUC) across CNN-based and PCNN-based models with statistically significant gains (p-values reported).
• The method is shown to boost performance when used with existing models, indicating its model-agnostic applicability as a plug-and-play component.
• Examples illustrate detection of false positives and distribution of removed samples across relations, aligning with observed dataset noise characteristics.