[Paper Review] Graph Neural Networks for Text Classification: A Survey

Kunze Wang, Yihao Ding+1 more|arXiv (Cornell University)|Apr 23, 2023

Topic Modeling140 references4 citations

TL;DR

This survey provides a comprehensive overview of graph neural network (GNN)-based text classification methods up to 2023, categorizing them into corpus-level and document-level GNNs. It analyzes graph construction techniques, model architectures, benchmark performance, and identifies key challenges and future directions, including integration with pre-trained models and dynamic graph structures.

ABSTRACT

Text Classification is the most essential and fundamental problem in Natural Language Processing. While numerous recent text classification models applied the sequential deep learning technique, graph neural network-based models can directly deal with complex structured text data and exploit global information. Many real text classification applications can be naturally cast into a graph, which captures words, documents, and corpus global features. In this survey, we bring the coverage of methods up to 2023, including corpus-level and document-level graph neural networks. We discuss each of these methods in detail, dealing with the graph construction mechanisms and the graph-based learning process. As well as the technological survey, we look at issues behind and future directions addressed in text classification using graph neural networks. We also cover datasets, evaluation metrics, and experiment design and present a summary of published performance on the publicly available benchmarks. Note that we present a comprehensive comparison between different techniques and identify the pros and cons of various evaluation metrics in this survey.

Motivation & Objective

• To provide a comprehensive, up-to-date survey of GNN-based text classification methods published through 2023.
• To systematically categorize and compare corpus-level and document-level GNN models with subcategories and architectural insights.
• To analyze graph construction mechanisms, including static and dynamic edge formation, and their impact on model performance.
• To summarize publicly available benchmarks, evaluation metrics, and model performance across datasets like 20NG, R8, R52, Ohsumed, and MR.
• To identify open challenges and future research directions, including integration with pre-trained models and scalability of inductive learning.

Proposed method

• Categorizes GNN-based text classification into two main paradigms: corpus-level GNNs (global, transductive) and document-level GNNs (local, inductive).
• Reviews graph construction techniques, including similarity-based, dependency parsing, and attention-based edge creation for both corpus- and document-level graphs.
• Analyzes model architectures such as TG-Transformer, HyperGAT, TextING, and TextSSL, focusing on message-passing mechanisms and graph pooling strategies.
• Compares performance across models using standard metrics like accuracy, F1, and macro-F1 on widely used benchmarks including 20NG, R8, R52, Ohsumed, and MR.
• Evaluates the impact of external resources (e.g., GloVe, BERT, RoBERTa) on model accuracy and generalization.
• Discusses limitations of current approaches, including reliance on static edges and scalability issues in corpus-level GNNs, and proposes dynamic and heterogeneous graph extensions.

Experimental results

Research questions

• RQ1 How do corpus-level and document-level GNNs differ in their graph construction and learning mechanisms for text classification?
• RQ2 What is the performance impact of using pre-trained word embeddings (e.g., GloVe) versus large language models (e.g., BERT) in GNN-based text classification?
• RQ3 Why do corpus-level GNNs generally outperform document-level GNNs on topic classification datasets but not on sentiment analysis datasets like MR?
• RQ4 What are the key challenges in graph construction for GNNs in text classification, particularly regarding edge dynamics and hypergraph modeling?
• RQ5 How can GNNs be effectively combined with pre-trained models and inductive learning to improve scalability and real-world deployment?

Key findings

• Corpus-level GNN models, which use transductive learning and global graph structures, generally outperform document-level GNNs on topic classification datasets such as 20NG, R8, and R52.
• On the MR sentiment analysis dataset, document-level GNNs perform comparably or better than corpus-level models due to the importance of word order, which corpus-level models often fail to capture.
• Models using BERT or RoBERTa achieve state-of-the-art performance, with TextING achieving 98.13% accuracy on R8 and 80.19% on MR, significantly outperforming GloVe-based models.
• Corpus-level GNNs using GloVe embeddings achieve 97.93% accuracy on 20NG, while document-level models like TextING achieve 97.8% on the same dataset, indicating strong performance even without pre-training.
• The use of dynamic or hyperedges in graph construction can improve modeling of complex word and document relationships, though most current models rely on static, statistics-based edges.
• Despite strong performance, corpus-level GNNs face scalability and deployment challenges due to large memory requirements and lack of incremental training support, highlighting a need for more efficient inductive variants.