[Paper Review] Relation Classification via Recurrent Neural Network

Dongxu Zhang, Dong Wang|arXiv (Cornell University)|Aug 5, 2015

Topic Modeling23 references283 citations

TL;DR

This paper proposes a recurrent neural network (RNN)-based framework for sentence-level relation classification, leveraging bidirectional long short-term memory networks to model long-distance dependencies between nominal pairs. Experiments show the RNN model significantly outperforms CNN-based models, especially on long-context relations, due to its superior ability to capture sequential and directional semantic patterns across extended spans of text.

ABSTRACT

Deep learning has gained much success in sentence-level relation classification. For example, convolutional neural networks (CNN) have delivered competitive performance without much effort on feature engineering as the conventional pattern-based methods. Thus a lot of works have been produced based on CNN structures. However, a key issue that has not been well addressed by the CNN-based method is the lack of capability to learn temporal features, especially long-distance dependency between nominal pairs. In this paper, we propose a simple framework based on recurrent neural networks (RNN) and compare it with CNN-based model. To show the limitation of popular used SemEval-2010 Task 8 dataset, we introduce another dataset refined from MIMLRE(Angeli et al., 2014). Experiments on two different datasets strongly indicates that the RNN-based model can deliver better performance on relation classification, and it is particularly capable of learning long-distance relation patterns. This makes it suitable for real-world applications where complicated expressions are often involved.

Motivation & Objective

• To address the limitation of CNN-based models in capturing long-distance dependencies in relation classification.
• To propose a simple yet effective RNN-based framework that models sequential context between nominal pairs.
• To validate the superiority of RNNs over CNNs on both the SemEval-2010 Task 8 dataset and a refined MIML-RE dataset.
• To empirically analyze the semantic accumulation process in RNNs versus CNNs, highlighting temporal modeling advantages.
• To demonstrate that position indicators (PI) are more effective than position features (PF) for encoding nominal positions.

Proposed method

• Uses a bidirectional LSTM network to encode the entire sentence context, capturing both forward and backward dependencies around the target nominals.
• Applies max-pooling over the hidden states of the RNN to generate a fixed-length sentence-level representation.
• Incorporates position indicators (PI) to explicitly encode the relative positions of the two nominals, improving modeling of directional relations.
• Processes input sequences word-by-word, allowing the model to accumulate semantic meaning progressively across long spans.
• Employs a simple feedforward layer on top of the pooled representation for final classification into predefined relation types.
• Uses word embeddings as input features, enabling end-to-end learning without manual feature engineering.

Experimental results

Research questions

• RQ1 Can an RNN-based model outperform CNN-based models in capturing long-distance dependency patterns in relation classification?
• RQ2 How does the RNN model’s sequential processing capability affect its performance on sentences with long contextual spans?
• RQ3 Is the position indicator (PI) approach more effective than the position feature (PF) method for encoding nominal positions in relation classification?
• RQ4 To what extent does the distribution of context lengths in benchmark datasets like SemEval-2010 limit the fair evaluation of model capabilities?
• RQ5 How do the semantic contributions of individual words differ between RNN and CNN models in forming sentence-level representations?

Key findings

• The RNN-based model achieves 79.6% F1 on the SemEval-2010 Task 8 dataset, outperforming the CNN baseline (77.4% F1), with a more pronounced advantage on long-context relations.
• On the MIML-RE-based refined dataset, the RNN model shows a distinct performance gain over CNNs, confirming its superiority in handling complex, long-distance patterns.
• The RNN model produces a smoother semantic contribution distribution across words, with a lower variance (0.0017) compared to CNNs (0.0025), indicating more coherent semantic accumulation.
• The RNN model correctly identifies the relation 'Instrument-Agency' in a sentence where the CNN model fails, due to its ability to capture the full context between 'witch' and 'magic'.
• The analysis confirms that RNNs are better at modeling directional and sequential dependencies, especially when the relevant words are far apart in the sentence.
• The position indicator (PI) method is empirically shown to be more universal and effective than the position feature (PF) method for encoding nominal positions.