[Paper Review] Classifying Relations by Ranking with Convolutional Neural Networks

Cícero Nogueira dos Santos, Bing Xiang+1 more|arXiv (Cornell University)|Apr 24, 2015

Topic Modeling12 references113 citations

TL;DR

This paper proposes CR-CNN, a convolutional neural network that classifies relations by ranking instead of softmax classification, using learned class embeddings and a novel pairwise ranking loss to reduce the impact of artificial 'Other' classes. On the SemEval-2010 Task 8 dataset, CR-CNN achieves an F1 score of 84.1 without handcrafted features, outperforming prior state-of-the-art methods by leveraging only word embeddings and the text between target nominals.

ABSTRACT

Relation classification is an important semantic processing task for which state-ofthe-art systems still rely on costly handcrafted features. In this work we tackle the relation classification task using a convolutional neural network that performs classification by ranking (CR-CNN). We propose a new pairwise ranking loss function that makes it easy to reduce the impact of artificial classes. We perform experiments using the the SemEval-2010 Task 8 dataset, which is designed for the task of classifying the relationship between two nominals marked in a sentence. Using CRCNN, we outperform the state-of-the-art for this dataset and achieve a F1 of 84.1 without using any costly handcrafted features. Additionally, our experimental results show that: (1) our approach is more effective than CNN followed by a softmax classifier; (2) omitting the representation of the artificial class Other improves both precision and recall; and (3) using only word embeddings as input features is enough to achieve state-of-the-art results if we consider only the text between the two target nominals.

Motivation & Objective

• To address the limitations of state-of-the-art relation classification systems that rely on costly, handcrafted features.
• To improve performance on the SemEval-2010 Task 8 benchmark by replacing softmax-based classification with a ranking-based approach.
• To reduce the negative impact of artificial 'Other' relation classes through a novel loss function and embedding omission strategy.
• To demonstrate that word embeddings alone, especially from the text between target nominals, are sufficient for state-of-the-art performance.
• To provide interpretability by identifying the most representative trigrams contributing to each relation class prediction.

Proposed method

• CR-CNN uses a convolutional neural network to generate a distributed representation of the input sentence from word embeddings.
• Each relation class is represented by a learned vector in a class embedding matrix, and the network computes a score for each class via dot product between the sentence representation and class embeddings.
• A novel pairwise ranking loss function is introduced to optimize the model by ranking correct relations higher than incorrect ones, reducing the influence of the artificial 'Other' class.
• The model is trained end-to-end using only word embeddings as input, with optional word position embeddings (WPEs) for improved performance.
• The network architecture enables interpretation by computing contribution values for trigrams based on their impact on the final score.
• The 'Other' class is omitted from the class embedding matrix during inference to improve precision and recall.

Experimental results

Research questions

• RQ1 Can a CNN-based model achieve state-of-the-art performance on relation classification without relying on expensive handcrafted features?
• RQ2 Does a ranking-based classification approach outperform traditional softmax-based classification in relation classification tasks?
• RQ3 How effective is the proposed pairwise ranking loss in mitigating the negative impact of artificial 'Other' relation classes?
• RQ4 To what extent can performance be maintained using only the text between the two target nominals, without additional features like position embeddings?
• RQ5 Can the model's predictions be interpreted by identifying the most influential trigrams in the input sentence?

Key findings

• CR-CNN achieves an F1 score of 84.1 on the SemEval-2010 Task 8 dataset, surpassing the previous state-of-the-art without using any handcrafted features.
• The ranking-based approach with the proposed loss function outperforms a standard CNN followed by a softmax classifier, demonstrating the effectiveness of the ranking paradigm.
• Omitting the 'Other' class embedding improves both precision and recall, indicating that artificial classes can distort model learning when treated equally.
• Using only word embeddings from the text between the two target nominals yields strong performance, showing that minimal input can be sufficient for high accuracy.
• The model identifies informative trigrams such as 'away from the' for Entity-Origin and 'the source of' for Origin-Entity, confirming interpretability and alignment with semantic meaning.
• The contribution analysis reveals that trigrams with high impact on the score are semantically relevant to the predicted relation type, validating the model's interpretability.