[Paper Review] Few-shot Relation Extraction via Bayesian Meta-learning on Relation Graphs

Meng Qu, Tianyu Gao+2 more|arXiv (Cornell University)|Jul 5, 2020

Topic ModelingComputer Science33 references65 citations

TL;DR

The paper proposes a Bayesian meta-learning framework that uses a global relation graph to model prior knowledge and performs posterior sampling of relation prototypes via stochastic gradient Langevin dynamics for few-shot and zero-shot relation extraction.

ABSTRACT

This paper studies few-shot relation extraction, which aims at predicting the relation for a pair of entities in a sentence by training with a few labeled examples in each relation. To more effectively generalize to new relations, in this paper we study the relationships between different relations and propose to leverage a global relation graph. We propose a novel Bayesian meta-learning approach to effectively learn the posterior distribution of the prototype vectors of relations, where the initial prior of the prototype vectors is parameterized with a graph neural network on the global relation graph. Moreover, to effectively optimize the posterior distribution of the prototype vectors, we propose to use the stochastic gradient Langevin dynamics, which is related to the MAML algorithm but is able to handle the uncertainty of the prototype vectors. The whole framework can be effectively and efficiently optimized in an end-to-end fashion. Experiments on two benchmark datasets prove the effectiveness of our proposed approach against competitive baselines in both the few-shot and zero-shot settings.

Motivation & Objective

• Motivate few-shot relation extraction with limited labeled data.
• Leverage a global relation graph to capture relationships between different relations.
• Model the prototype vectors as posteriors rather than point estimates to handle uncertainty.
• Use graph neural networks to parameterize priors from the relation graph and apply Langevin dynamics for posterior sampling.
• Demonstrate effectiveness on FewRel and NYT-25 in both few-shot and zero-shot settings.

Proposed method

• Represent each relation with a prototype vector and treat it as a random variable with a posterior distribution.
• Parameterize the prior p(v_T|G) using a graph neural network on a global relation graph derived from Wikidata embeddings.
• Define the likelihood on support sentences p(y_S|x_S, v_T) using a softmax over prototype vectors and a sentence encoder E.
• Approximate the intractable posterior p(v_T|x_S,y_S,G) via Monte Carlo sampling with stochastic gradient Langevin dynamics to draw multiple prototype samples.
• Use the sampled prototypes to compute p(y_Q|x_Q, v_T) for query predictions and optimize end-to-end.
• Initialize Langevin samples with a theoretically motivated scheme that combines graph-based priors and support-set encodings to accelerate convergence.

Experimental results

Research questions

• RQ1 Can a global relation graph provide transferable priors to improve few-shot relation extraction across diverse relations?
• RQ2 Does Bayesian meta-learning with posterior prototype vectors outperform point-estimate meta-learning methods in few-shot and zero-shot settings?
• RQ3 What is the impact of incorporating a graph-based prior on the task Generalization and zero-shot capability?
• RQ4 How effective is stochastic gradient Langevin dynamics at sampling from complex posteriors for relation prototypes?

Key findings

• On FewRel test set, REGRAB achieves strong performance, e.g., 5-Way 1-Shot: 90.30% and 5-Way 5-Shot: 94.25%.
• On NYT-25 test set, REGRAB achieves 5-Way 1-Shot: 89.76%, 5-Way 5-Shot: 95.66%, 10-Way 1-Shot: 84.11%, and 10-Way 5-Shot: 92.48%.
• Ablation shows removing the graph-based prior reduces accuracy (FewRel validation, 5-Way 1-Shot: 87.95 vs 85.82; 10-Way 1-Shot: 80.26 vs 77.70).
• Zero-shot performance remains effective when using the global relation graph to compute priors, outperforming non-graph baselines in several settings.