[Paper Review] SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals

Iris Hendrickx, Su‐Nam Kim+7 more|arXiv (Cornell University)|Nov 23, 2019

Biomedical Text Mining and OntologiesBiochemistry, Genetics and Molecular Biology5 references447 citations

TL;DR

Introduces a multi-way classification task for semantic relations between nominal pairs, provides dataset creation, annotation guidelines, and an extensive evaluation of 28 systems from 10 teams. UTD achieved the top performance with over 82% F1 on the full training set.

ABSTRACT

In response to the continuing research interest in computational semantic analysis, we have proposed a new task for SemEval-2010: multi-way classification of mutually exclusive semantic relations between pairs of nominals. The task is designed to compare different approaches to the problem and to provide a standard testbed for future research. In this paper, we define the task, describe the creation of the datasets, and discuss the results of the participating 28 systems submitted by 10 teams.

Motivation & Objective

• Define a standardized multi-way (9+1) semantic relation classification task for nominal pairs.
• Create a large, annotated dataset with training and test splits for robust evaluation.
• Provide annotation guidelines and an evaluation metric to compare diverse approaches.
• Assess system performance across varying training data sizes to understand data vs. model effects.

Proposed method

• Assemble an inventory of nine mutually exclusive semantic relations plus an Other category with careful guidance to minimize overlap.
• Follow a three-round annotation process to obtain high-quality labeled instances and compute inter-annotator agreement.
• Release a training set (8,000 examples) and a test set (1,717 examples incl. 891 re-annotated from SemEval-1 Task 4).
• Provide a detailed scorer including confusion matrix, accuracy, precision, recall, and F1 (micro and macro) with macro-F1 over the (9+1) relations.
• Use direction-aware evaluation for the final macro-F1 metric.
• Encourage varying training data sizes (TD1–TD4) to study data effects on performance.

Experimental results

Research questions

• RQ1 Can a standardized 10-way (9 relations + Other) labeling scheme reliably discriminate between nominal-pair semantic relations?
• RQ2 What is the impact of data quantity on classifier performance across diverse systems?
• RQ3 How do different model architectures and resources affect performance on a uniform task and dataset?
• RQ4 Which relations are easiest or hardest to classify, and why?
• RQ5 Does system combination (ensembling) outperform the best single system?

Key findings

• The best system (UTD) achieved over 82% macro-F1 on TD4, outperforming the second-best by more than 4 percentage points.
• There is a substantial gain from TD1 to TD4 across all systems, with diminishing returns from TD3 to TD4 for many systems.
• Multi-label vs binary classifiers: some architectures benefit from multi-way classification (including Other), while others rely on binary strategies; TD4 performance differences between approaches are system-dependent.
• CE (Cause-Effect) was generally the easiest relation to classify; IA (Instrument-Agency) and PP (Product-Producer) were among the hardest, with IA also showing high annotation agreement in some setups.
• System combination via majority vote provided limited improvement over the top system, suggesting the top performer’s margin may be hard to beat with simple ensembles.
• Several systems benefited from richer resources (WordNet, Google n-grams, Cyc), but integration complexity makes gains inconsistent across setups.
• There were 152 consistently misclassified instances across all systems, indicating limits of shallow feature-based approaches and the need for deeper lexical reasoning.