[Paper Review] QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering

Michihiro Yasunaga, Hongyu Ren+3 more|arXiv (Cornell University)|Apr 13, 2021

Topic Modeling62 references49 citations

TL;DR

QA-GNN jointly reasons over a QA context and a knowledge graph by building a working graph with a QA context node, applying relevance scoring with a language model, and performing attention-based graph neural network reasoning to answer questions.

ABSTRACT

The problem of answering questions using knowledge from pre-trained language models (LMs) and knowledge graphs (KGs) presents two challenges: given a QA context (question and answer choice), methods need to (i) identify relevant knowledge from large KGs, and (ii) perform joint reasoning over the QA context and KG. In this work, we propose a new model, QA-GNN, which addresses the above challenges through two key innovations: (i) relevance scoring, where we use LMs to estimate the importance of KG nodes relative to the given QA context, and (ii) joint reasoning, where we connect the QA context and KG to form a joint graph, and mutually update their representations through graph neural networks. We evaluate our model on QA benchmarks in the commonsense (CommonsenseQA, OpenBookQA) and biomedical (MedQA-USMLE) domains. QA-GNN outperforms existing LM and LM+KG models, and exhibits capabilities to perform interpretable and structured reasoning, e.g., correctly handling negation in questions.

Motivation & Objective

• Motivate the problem of answering questions with both pre-trained language models and knowledge graphs, highlighting limitations of LM-only and KG-only approaches.
• Propose a unified LM+KG model that jointly reasons over QA context and KG through a joint graph structure.
• Introduce KG node relevance scoring to prune irrelevant KG nodes conditioned on the QA context.
• Develop an attention-based GNN on a working graph that updates QA context and KG representations in tandem.

Proposed method

• Encode the QA context with a pre-trained LM and retrieve a KG subgraph around question/answer entities.
• Construct a working graph by introducing a QA context node and linking it to topic KG entities with new relations, enabling joint reasoning.
• Compute KG node relevance scores by concatenating the QA context with KG entity text and evaluating likelihood with the LM.
• Use a multi-layer graph attention network on the working graph with node type, relation, and relevance-aware messages and attention to update representations.
• Predict the answer by combining LM QA context embedding, GNN-derived representations, and a pooled KG representation through an MLP.
• Discuss the computational complexity and justify the use of GNNs for this reasoning task.

Experimental results

Research questions

• RQ1 Can a joint LM+KG model outperform LM-only and existing LM+KG baselines on varied QA domains requiring knowledge-based reasoning?
• RQ2 Does relevance-scored KG subgraph and joint QA context–KG reasoning improve performance on structured reasoning tasks such as handling negation?
• RQ3 How does QA-GNN perform interpretability through attention on the working graph and provide reasoning traces?
• RQ4 Is the approach effective across commonsense and biomedical QA domains with different KG sources?

Key findings

• QA-GNN outperforms fine-tuned LMs and prior LM+KG models across CommonsenseQA, OpenBookQA, and MedQA-USMLE.
• QA-GNN particularly improves performance on questions requiring structured reasoning, such as negation handling, with up to 4.6% absolute gain over the LM baseline in CommonsenseQA's negation cases.
• Ablations show that joint graph construction (QA context link to KG) and KG node relevance scoring both contribute to performance gains.
• QA-GNN enables extraction of reasoning subgraphs as explanations beyond single-path traces, enabling broader interpretability.
• On CommonsenseQA, QA-GNN achieves 76.54% IHdev and 73.41% IHtest accuracy, surpassing MHGRN by 2.3 percentage points in IHtest.
• On OpenBookQA and MedQA-USMLE, QA-GNN attains competitive results, matching or exceeding baselines under the same LM.