[Paper Review] ETH-DS3Lab at SemEval-2018 Task 7: Effectively Combining Recurrent and Convolutional Neural Networks for Relation Classification and Extraction

Jonathan Rotsztejn, Nora Hollenstein+1 more|arXiv (Cornell University)|Apr 5, 2018

Natural Language Processing Techniques18 references3 citations

TL;DR

This paper presents a hybrid deep learning system combining convolutional (CNN) and recurrent (RNN) neural networks with domain-specific word embeddings and extensive preprocessing to achieve state-of-the-art performance in relation classification and extraction on SemEval-2018 Task 7. The ensemble model ranked first in three out of four subtasks, with an F1 score of 81.7% on clean data and 90.4% on noisy data, demonstrating the effectiveness of combining CNN and RNN architectures with length-aware fusion and rule-based post-processing.

ABSTRACT

Reliably detecting relevant relations between entities in unstructured text is a valuable resource for knowledge extraction, which is why it has awaken significant interest in the field of Natural Language Processing. In this paper, we present a system for relation classification and extraction based on an ensemble of convolutional and recurrent neural networks that ranked first in 3 out of the 4 subtasks at SemEval 2018 Task 7. We provide detailed explanations and grounds for the design choices behind the most relevant features and analyze their importance.

Motivation & Objective

• To develop a robust neural network system for relation classification and extraction in scientific literature, where relations are highly specialized and context-dependent.
• To improve performance on both clean and noisy data by combining the strengths of convolutional and recurrent architectures.
• To address challenges in long-sequence relation detection by incorporating length-aware weighting and sentence cropping.
• To enhance model generalization through domain-specific word embeddings and data augmentation techniques.
• To ensure consistency with annotation schemes via rule-based post-processing for symmetrical and overlapping relations.

Proposed method

• The system employs an ensemble of two independent models: a 1D-CNN with multiple filter widths and max-pooling, and a bidirectional LSTM network with forward and backward encoding.
• Word embeddings are trained on 91 million tokens from scientific NLP papers (ArXiv and ACL ARC) and fine-tuned for the task, with 200 dimensions selected as optimal.
• Sentences are cropped to include only the span between entities, and a maximum length threshold of 19 words is applied to reduce false positives in long sentences.
• A dynamic, length-dependent fusion strategy assigns higher weights to RNN predictions in longer sequences using the formula $ w_{rnn,i} = 0.5 + \text{sign}(s_i) \cdot s_i^2 $, where $ s_i $ normalizes sentence length.
• Model outputs are averaged across multiple random seeds to form a diverse ensemble, improving generalization and robustness.
• Post-processing enforces annotation consistency by rejecting invalid COMPARE predictions and resolving overlapping entity relations via conflict resolution (shorter relations first, then class frequency, then random).

Experimental results

Research questions

• RQ1 How effective is the combination of CNN and RNN architectures for relation classification in scientific text compared to individual models?
• RQ2 To what extent do domain-specific word embeddings and data preprocessing improve performance on low-resource, noisy relation extraction tasks?
• RQ3 Can length-aware fusion of CNN and RNN predictions enhance performance on long sequences without overfitting?
• RQ4 How do data augmentation and sentence reversal impact model generalization in low-resource relation classification scenarios?
• RQ5 What role does ensemble learning and post-processing play in achieving state-of-the-art results on SemEval-2018 Task 7?

Key findings

• The ensemble model achieved an F1 score of 81.7% on Subtask 1.1 (clean data), ranking first among 14 teams.
• On Subtask 1.2 (noisy data), the system achieved 90.4% F1, the highest score among all submissions.
• For joint relation extraction and classification (Subtask 2.C), the model reached 49.3% F1, ranking first.
• The system ranked second in relation extraction only (Subtask 2.E) with 48.8% F1, demonstrating strong performance on the more challenging extraction task.
• The feature ablation study showed that data combination (CSD), sentence reversal (RS), and data generation (GD) each contributed significantly to performance gains, independent of the model type.
• The length-aware fusion strategy improved performance on longer sequences, confirming that RNNs are more effective than CNNs for long-range dependencies in this context.