[Paper Review] FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation

Xu Han, Hao Zhu+5 more|arXiv (Cornell University)|Oct 24, 2018

Topic Modeling20 references43 citations

TL;DR

The paper introduces FewRel, a large-scale, human-annotated few-shot relation classification dataset with 100 relations and 70k sentences, and evaluates multiple few-shot learning models on it, showing humans significantly outperform current models.

ABSTRACT

We present a Few-Shot Relation Classification Dataset (FewRel), consisting of 70, 000 sentences on 100 relations derived from Wikipedia and annotated by crowdworkers. The relation of each sentence is first recognized by distant supervision methods, and then filtered by crowdworkers. We adapt the most recent state-of-the-art few-shot learning methods for relation classification and conduct a thorough evaluation of these methods. Empirical results show that even the most competitive few-shot learning models struggle on this task, especially as compared with humans. We also show that a range of different reasoning skills are needed to solve our task. These results indicate that few-shot relation classification remains an open problem and still requires further research. Our detailed analysis points multiple directions for future research. All details and resources about the dataset and baselines are released on http://zhuhao.me/fewrel.

Motivation & Objective

• Create a large, high-quality few-shot relation classification dataset to address noise in distant supervision.
• Evaluate modern few-shot learning methods on RC to establish baselines and identify gaps to human performance.
• Analyze the diversity of expressions for relations and suggest directions for future research.
• Provide resources and baselines to advance few-shot learning in NLP tasks.

Proposed method

• Construct FewRel by distantly supervising candidate sentences from Wikipedia and Wikidata, then crowd-filter for accuracy.
• Limit each relation’s candidate set to one instance per entity pair to reduce memorization of pairs.
• Sample 700 positive instances per relation across 100 relations after filtering.
• Adopt standard N-way K-shot settings and adapt multiple state-of-the-art few-shot methods to RC.
• Represent sentences with CNN or PCNN encoders, and evaluate baselines like Finetune and kNN, plus meta-learning and graph-based approaches.
• Release dataset and baselines publicly for reproducibility.

Experimental results

Research questions

• RQ1 Can few-shot learning methods achieve competitive accuracy on RC when trained on a large, high-quality FewRel dataset?
• RQ2 How do state-of-the-art few-shot models (e.g., prototypical networks, GNNs, SNAIL, meta-networks) perform on RC compared to humans?
• RQ3 What types of reasoning are required to correctly classify relations in FewRel (pattern-based, common-sense, logical, co-reference)?
• RQ4 What are the limitations of current RC models under few-shot settings and where can future research improve?

Key findings

• FewRel contains 100 relations with 700 instances each, totaling 70,000 instances across 100 relations.
• State-of-the-art few-shot models outperform vanilla CNN/PCNN baselines but still lag behind human performance on FewRel.
• CNN-based few-shot models with advanced methods achieve higher accuracies than Finetune or kNN baselines in 5-way/10-way settings.
• Prototypical Networks and related approaches reach the best CNN-based results among evaluated models in several configurations.
• Humans achieve substantially higher accuracy (e.g., 92.22% in 5-way 1-shot) than models, indicating room for improvement.