[Paper Review] Graph Neural Networks for Natural Language Processing: A Survey

Lingfei Wu, Yu Chen+6 more|arXiv (Cornell University)|Jun 10, 2021

Topic ModelingComputer Science421 references68 citations

TL;DR

This survey provides a comprehensive taxonomy and review of Graph Neural Networks (GNNs) for NLP, covering graph construction, representation learning, encoder-decoder models, applications, datasets, and future directions.

ABSTRACT

Deep learning has become the dominant approach in coping with various tasks in Natural LanguageProcessing (NLP). Although text inputs are typically represented as a sequence of tokens, there isa rich variety of NLP problems that can be best expressed with a graph structure. As a result, thereis a surge of interests in developing new deep learning techniques on graphs for a large numberof NLP tasks. In this survey, we present a comprehensive overview onGraph Neural Networks(GNNs) for Natural Language Processing. We propose a new taxonomy of GNNs for NLP, whichsystematically organizes existing research of GNNs for NLP along three axes: graph construction,graph representation learning, and graph based encoder-decoder models. We further introducea large number of NLP applications that are exploiting the power of GNNs and summarize thecorresponding benchmark datasets, evaluation metrics, and open-source codes. Finally, we discussvarious outstanding challenges for making the full use of GNNs for NLP as well as future researchdirections. To the best of our knowledge, this is the first comprehensive overview of Graph NeuralNetworks for Natural Language Processing.

Motivation & Objective

• Propose a new taxonomy of GNNs for NLP organizing research along graph construction, graph representation learning, and graph-based encoder-decoder models.
• Summarize state-of-the-art GNN-based approaches for diverse NLP tasks with detailed comparisons of graph construction and learning techniques.
• Provide an inventory of NLP applications leveraging GNNs, including datasets, metrics, and open-source codes.
• Discuss outstanding challenges and outline future research directions for fully leveraging GNNs in NLP.

Proposed method

• Introduce a taxonomy of GNNs for NLP across four axes: graph construction, graph representation learning, encoder-decoder models, and applications.
• Review graph construction approaches including static and dynamic graphs for NLP tasks.
• Survey graph representation learning techniques, detailing spectral, spatial, attention-based, and recurrent graph filters.
• Describe graph-based encoder-decoder models such as graph2seq, graph2tree, and graph2graph, and their applicability to NLP.
• Summarize NLP applications, benchmark datasets, evaluation metrics, and available open-source codes.
• Discuss challenges and propose future research directions for improving graph construction, learning, and model effectiveness in NLP.

Figure 1: The taxonomy, which systematically organizes GNNs for NLP along four axes: graph construction, graph representation learning, encoder-decoder models, and the applications.

Experimental results

Research questions

• RQ1 What graph constructions are most effective for various NLP tasks and how can text data be transformed into informative graph structures?
• RQ2 Which GNN variants and graph filters best suit different graph structures (static/dynamic, directed/undirected, multi-relational, heterogeneous) for NLP problems?
• RQ3 How can graph-based encoders and decoders (graph2seq, graph2tree, graph2graph) be optimally applied to diverse NLP tasks, and what datasets and metrics best reflect their performance?
• RQ4 What are the key challenges and open research directions in applying GNNs to NLP, and how can they be addressed in future work.

Key findings

• The paper provides the first comprehensive overview of Graph Neural Networks applied to Natural Language Processing.
• It offers a detailed taxonomy and comparative discussion of graph construction, representation learning, and graph-based encoder-decoder models for NLP.
• It compiles numerous NLP applications, benchmarking datasets, evaluation metrics, and open-source codes.
• It discusses practical challenges in automatic graph construction, suitable GNN variants for different graph types, and modeling complex NLP data.
• It outlines future research directions and opportunities to advance the use of GNNs in NLP.

Figure 2: An example is shown for the dependency graph (left) and the constituency graph (right), respectively. The text input is from JOBS640 (Luke, 2005 ) dataset.