[Paper Review] BertGCN: Transductive Text Classification by Combining GCN and BERT

Yuxiao Lin, Yuxian Meng+5 more|arXiv (Cornell University)|May 12, 2021

Text and Document Classification TechnologiesComputer Science43 references46 citations

TL;DR

BertGCN jointly trains a BERT encoder and a GCN on a heterogeneous word-document graph to perform transductive text classification, achieving state-of-the-art results on multiple datasets.

ABSTRACT

In this work, we propose BertGCN, a model that combines large scale pretraining and transductive learning for text classification. BertGCN constructs a heterogeneous graph over the dataset and represents documents as nodes using BERT representations. By jointly training the BERT and GCN modules within BertGCN, the proposed model is able to leverage the advantages of both worlds: large-scale pretraining which takes the advantage of the massive amount of raw data and transductive learning which jointly learns representations for both training data and unlabeled test data by propagating label influence through graph convolution. Experiments show that BertGCN achieves SOTA performances on a wide range of text classification datasets. Code is available at https://github.com/ZeroRin/BertGCN.

Motivation & Objective

• Motivate combining large-scale pretraining with transductive learning for text classification.
• Propose a heterogeneous word-document graph with BERT-initialized document embeddings for GCN input.
• Jointly train BERT and GCN modules and optionally interpolate their predictions.
• Address training inefficiencies in GCNs on large graphs with a memory bank approach.

Proposed method

• Construct a heterogeneous word-document graph where document nodes use BERT embeddings as initial features.
• Use a two-layer GCN to propagate information over the normalized adjacency matrix.
• Interpolate BertGCN predictions with direct BERT predictions to form the final output.
• Train with a cross-entropy loss on labeled documents while updating both BERT and GCN parameters.
• Introduce a memory bank to store all document embeddings to decouple batch size from graph size during training.

Experimental results

Research questions

• RQ1 Can large-scale pretraining benefits be leveraged in transductive GCN-based text classification?
• RQ2 Does jointly training BERT and GCN improve performance over using either component alone?
• RQ3 Does interpolating BERT and BertGCN predictions yield better results than using BertGCN alone?
• RQ4 How can memory efficiency be achieved for training BertGCN on large graphs?

Key findings

• BertGCN achieves state-of-the-art performance across multiple datasets compared to TextGCN, SGC, BERT, and RoBERTa baselines.
• RoBERTa-based BertGCN variants reach the top scores on all tested datasets, with notable gains on longer documents like 20NG and Ohsumed.
• The best RoBERTaGCN performance on 20NG occurs around lambda=0.7, indicating a balance between GCN and BERT components.
• A memory bank enables efficient training by decoupling batch size from the total number of nodes in the graph.
• A small learning rate for the BERT module and finetuning before training improve stability and accuracy in the memory bank setup.