[Paper Review] SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning

Keyu Duan, Qian Liu+5 more|arXiv (Cornell University)|Aug 3, 2023

Advanced Graph Neural Networks6 citations

TL;DR

SimTeG proposes a simple yet effective method for textual graph learning by first fine-tuning a pre-trained language model via parameter-efficient fine-tuning (PEFT) on downstream tasks, then using the final hidden states as node features for any GNN. This approach achieves state-of-the-art performance—78.04% accuracy on OGBN-Arxiv—surpassing complex architectures with minimal modification.

ABSTRACT

Textual graphs (TGs) are graphs whose nodes correspond to text (sentences or documents), which are widely prevalent. The representation learning of TGs involves two stages: (i) unsupervised feature extraction and (ii) supervised graph representation learning. In recent years, extensive efforts have been devoted to the latter stage, where Graph Neural Networks (GNNs) have dominated. However, the former stage for most existing graph benchmarks still relies on traditional feature engineering techniques. More recently, with the rapid development of language models (LMs), researchers have focused on leveraging LMs to facilitate the learning of TGs, either by jointly training them in a computationally intensive framework (merging the two stages), or designing complex self-supervised training tasks for feature extraction (enhancing the first stage). In this work, we present SimTeG, a frustratingly Simple approach for Textual Graph learning that does not innovate in frameworks, models, and tasks. Instead, we first perform supervised parameter-efficient fine-tuning (PEFT) on a pre-trained LM on the downstream task, such as node classification. We then generate node embeddings using the last hidden states of finetuned LM. These derived features can be further utilized by any GNN for training on the same task. We evaluate our approach on two fundamental graph representation learning tasks: node classification and link prediction. Through extensive experiments, we show that our approach significantly improves the performance of various GNNs on multiple graph benchmarks.

Motivation & Objective

• To investigate whether simple, effective use of pre-trained language models can significantly improve textual graph representation learning.
• To address the underutilization of modern language models in the feature extraction stage of graph learning, which still relies on traditional methods like BoW.
• To evaluate whether language model features enhance performance not only in node classification but also in link prediction, a task more reliant on structural information.
• To determine whether the choice of language model backbone and fine-tuning strategy affects downstream GNN performance.
• To demonstrate that a simple, modular approach can outperform complex, end-to-end frameworks in textual graph learning.

Proposed method

• Fine-tune a pre-trained language model using parameter-efficient fine-tuning (PEFT) on the downstream task (e.g., node classification) with task-specific labels.
• Extract node representations from the last hidden layer of the fine-tuned language model, discarding the classification head.
• Use these derived text embeddings as input features for any Graph Neural Network (GNN) in the downstream graph learning task.
• Train the GNN on the same task using the language model-generated features, maintaining a two-stage paradigm: feature extraction followed by graph learning.
• Evaluate performance on node classification and link prediction across multiple benchmarks, including OGBN-Arxiv, OGBN-Products, and OGBN-Citation2.
• Compare results with full fine-tuning and traditional feature engineering, analyzing overfitting and model sensitivity.

Experimental results

Research questions

• RQ1 How much can language model features improve GNN performance on textual graphs, and is the improvement consistent across different GNN architectures?
• RQ2 Which type of pre-trained language model—retrieval-optimized or masked language modeling (MLM)—yields better node representations for textual graph learning?
• RQ3 Does incorporating language model features improve performance on link prediction, a task that emphasizes structural patterns over text?
• RQ4 How sensitive is GNN training to the choice of language model backbone and fine-tuning method?
• RQ5 Can a simple, modular approach outperform complex, end-to-end frameworks in textual graph representation learning?

Key findings

• A simple two-layer GraphSAGE trained on SimTeG features achieves 77.48% accuracy on OGBN-Arxiv, matching state-of-the-art performance from more complex models.
• When combined with a SOTA GNN, SimTeG achieves a new state-of-the-art accuracy of 78.04% on OGBN-Arxiv.
• Parameter-efficient fine-tuning significantly reduces overfitting compared to full fine-tuning, as shown by a 2.52 vs. 9.11 delta in overfitting (Test Acc. - Train Acc.) on the OGBN-Arxiv node classification task.
• Retrieval-optimized language models like e5-large and all-roberta-large-v1 outperform standard MLM models like roberta-large when used as backbones in SimTeG.
• The performance ranking of language models (e5-large > all-roberta-large-v1 > all-MiniLM-L6-v2) aligns with their MTEB leaderboard scores, indicating that better retrieval models yield better graph representations.
• The method is robust and modular: it complements any GNN and language model, and performance scales with the quality of the underlying LM.