[Paper Review] Multiple Range-Restricted Bidirectional Gated Recurrent Units with Attention for Relation Classification

Jonggu Kim, Jong-Hyeok Lee|arXiv (Cornell University)|Jul 5, 2017

Topic Modeling15 references3 citations

TL;DR

This paper proposes a novel RNN-based model using multiple range-restricted bidirectional Gated Recurrent Units (GRUs) with attention for relation classification, focusing only on nominal and relation spans in a sentence. Without relying on external linguistic features, the model achieves an F1 score of 84.3% on the SemEval-2010 dataset, matching state-of-the-art performance.

ABSTRACT

Most of neural approaches to relation classification have focused on finding short patterns that represent the semantic relation using Convolutional Neural Networks (CNNs) and those approaches have generally achieved better performances than using Recurrent Neural Networks (RNNs). In a similar intuition to the CNN models, we propose a novel RNN-based model that strongly focuses on only important parts of a sentence using multiple range-restricted bidirectional layers and attention for relation classification. Experimental results on the SemEval-2010 relation classification task show that our model is comparable to the state-of-the-art CNN-based and RNN-based models that use additional linguistic information.

Motivation & Objective

• To improve relation classification performance using RNNs without relying on handcrafted linguistic features.
• To investigate whether restricting GRU layers to specific sentence spans (nominals and relation) enhances feature abstraction.
• To evaluate the effectiveness of attention mechanisms in focusing on relation-relevant spans in a sentence.
• To compare the performance of models that extract only nominals, only the relation, or both, under range restrictions.

Proposed method

• The model uses pretrained 100-dimensional word embeddings as input to represent each word in a sentence.
• Three separate bidirectional GRU layers are applied with range restrictions: one for the first nominal (e1), one for the second nominal (e2), and one for the relation span between them.
• A masking technique enforces range restrictions, ensuring each GRU processes only its designated span, improving focus on relevant syntactic and semantic patterns.
• Attention is applied to the final hidden states of the three GRUs to dynamically weigh their contributions based on relevance to the relation class.
• The final sentence representation is formed by concatenating the attended outputs of the three GRU layers.
• A feed-forward layer with softmax computes the final relation class scores using a ranking loss function to optimize for margin-based classification.

Experimental results

Research questions

• RQ1 Can RNN-based models achieve competitive performance in relation classification without using external linguistic features like POS tags or dependency paths?
• RQ2 Does restricting GRU layers to specific spans (nominal and relation) improve feature abstraction and classification accuracy?
• RQ3 How does the combination of range restriction and attention affect performance compared to standard bidirectional GRUs?
• RQ4 What is the relative contribution of nominal spans versus the relation span in predicting the correct relation class?

Key findings

• The proposed model achieved an F1 score of 84.3% on the SemEval-2010 test set using only pretrained word embeddings and no additional linguistic features.
• The model outperformed standard attention-based bidirectional GRUs (att-BGRU) when applied without range restrictions, indicating the benefit of span-specific modeling.
• The model that jointly abstracted both nominals and the relation span achieved the highest performance (84.3%), suggesting complementary information from all three components.
• The ablation study showed that restricting GRUs to only the relation span yielded a lower F1 (82.9%) than the full model, indicating that nominal features are crucial for performance.
• The model's performance was comparable to state-of-the-art CNN and RNN models that use additional linguistic resources such as POS tags, WordNet, and shortest dependency paths.