[Paper Review] Understanding Graph Convolutional Networks for Text Classification

Soyeon Caren Han, Zihan Yuan+3 more|arXiv (Cornell University)|Mar 30, 2022

Advanced Graph Neural Networks6 citations

TL;DR

This paper provides a comprehensive empirical analysis of graph construction and GCN learning mechanisms in text classification, evaluating node/edge representations and GCN layer depth across full and semi-supervised settings. It finds that using all three edge types (d2w+w2w+d2d) and two GCN layers yields optimal performance, with one-hot embeddings outperforming BERT in low-resource languages due to poor pretraining quality.

ABSTRACT

Graph Convolutional Networks (GCN) have been effective at tasks that have rich relational structure and can preserve global structure information of a dataset in graph embeddings. Recently, many researchers focused on examining whether GCNs could handle different Natural Language Processing tasks, especially text classification. While applying GCNs to text classification is well-studied, its graph construction techniques, such as node/edge selection and their feature representation, and the optimal GCN learning mechanism in text classification is rather neglected. In this paper, we conduct a comprehensive analysis of the role of node and edge embeddings in a graph and its GCN learning techniques in text classification. Our analysis is the first of its kind and provides useful insights into the importance of each graph node/edge construction mechanism when applied at the GCN training/testing in different text classification benchmarks, as well as under its semi-supervised environment.

Motivation & Objective

• To understand how graph construction—specifically node and edge design—affects GCN-based text classification performance.
• To investigate the impact of GCN layer depth on capturing neighborhood information for improved classification.
• To evaluate performance in both full-corpus and semi-supervised, low-resource settings where labeled data is scarce.
• To identify the most effective node feature representations (e.g., one-hot vs. BERT) under varying data availability and dataset size.
• To provide open-source code to enable reproducibility and further research in GCN-based text classification.

Proposed method

• Constructed corpus-level graphs using documents and words as nodes, with edges based on co-occurrence (d2w), word-word (w2w), and document-document (d2d) relationships.
• Evaluated multiple node feature representations: one-hot encoding and BERT-based contextual embeddings.
• Systematically varied GCN layer counts from 1 to 5 to assess the effect of neighborhood aggregation depth.
• Performed experiments on five standard text classification benchmarks (20NG, R8, R52, Ohsumed, MR) in a full-corpus setting.
• Extended evaluation to nine datasets including low-resource languages (Chinese, Korean, African) under semi-supervised, limited-label conditions.
• Used standard metrics (accuracy, macro F1, weighted F1) and visualized performance trends across training proportions.

Experimental results

Research questions

• RQ1 What is the optimal textual graph construction approach—specifically, which node and edge types—yields the best GCN performance in text classification?
• RQ2 How do different node feature representations (e.g., one-hot vs. BERT) affect GCN performance across datasets with varying numbers of classes and data availability?
• RQ3 What is the optimal number of GCN layers for capturing neighborhood information without over-smoothing or under-exploring?
• RQ4 How does performance vary in low-resource settings with minimal labeled data, and does the optimal configuration differ from full-data scenarios?
• RQ5 Does the performance of BERT embeddings degrade in low-resource languages compared to one-hot representations?

Key findings

• Using all three edge types—word-document (d2w), word-word (w2w), and document-document (d2d)—produced the highest accuracy across all datasets, significantly outperforming d2w-only graphs.
• The d2w-only graph performed worst, indicating that relying solely on word-document co-occurrence fails to capture sufficient global structural information.
• Two GCN layers consistently achieved the best performance across all five full-corpus benchmarks, with performance degrading when using fewer or more layers.
• In low-resource settings, two layers remained optimal overall, though some languages like Chinese showed a performance rebound at four layers, suggesting language-specific optimal depths.
• One-hot embeddings outperformed BERT embeddings on low-resource language datasets (e.g., ChSenti, Waimai, Xhosa), likely due to poor pretraining quality in low-resource languages.
• Small-category datasets (e.g., R8, MR) benefited more from BERT embeddings, while large-category datasets showed less improvement, indicating category size influences embedding utility.