[Paper Review] Transductive Linear Probing: A Novel Framework for Few-Shot Node Classification

Zhen Tan, Song Wang+3 more|arXiv (Cornell University)|Dec 11, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes Transductive Linear Probing (TLP), a novel framework for few-shot node classification that transfers pretrained node embeddings from graph contrastive learning (GCL) to fine-tune a linear classifier on few labeled nodes. Surprisingly, TLP with self-supervised GCL pretraining outperforms state-of-the-art meta-learning methods even without any ground-truth labels during pretraining, demonstrating its robustness and transferability in low-supervision settings.

ABSTRACT

Few-shot node classification is tasked to provide accurate predictions for nodes from novel classes with only few representative labeled nodes. This problem has drawn tremendous attention for its projection to prevailing real-world applications, such as product categorization for newly added commodity categories on an E-commerce platform with scarce records or diagnoses for rare diseases on a patient similarity graph. To tackle such challenging label scarcity issues in the non-Euclidean graph domain, meta-learning has become a successful and predominant paradigm. More recently, inspired by the development of graph self-supervised learning, transferring pretrained node embeddings for few-shot node classification could be a promising alternative to meta-learning but remains unexposed. In this work, we empirically demonstrate the potential of an alternative framework, extit{Transductive Linear Probing}, that transfers pretrained node embeddings, which are learned from graph contrastive learning methods. We further extend the setting of few-shot node classification from standard fully supervised to a more realistic self-supervised setting, where meta-learning methods cannot be easily deployed due to the shortage of supervision from training classes. Surprisingly, even without any ground-truth labels, transductive linear probing with self-supervised graph contrastive pretraining can outperform the state-of-the-art fully supervised meta-learning based methods under the same protocol. We hope this work can shed new light on few-shot node classification problems and foster future research on learning from scarcely labeled instances on graphs.

Motivation & Objective

• To address the limitations of meta-learning in few-shot node classification, particularly its reliance on abundant base-class labels and episodic training.
• To explore whether transferring pretrained node embeddings from graph contrastive learning (GCL) can serve as a viable alternative to meta-learning.
• To evaluate the performance of TLP under realistic self-supervised settings where meta-learning is infeasible due to label scarcity.
• To analyze the role of supervision in GCL pretraining for few-shot node classification.
• To provide empirical evidence that generalizable GNN encoders are key to few-shot node classification.

Proposed method

• TLP transfers node embeddings pretrained via graph contrastive learning (GCL) to a linear classifier fine-tuned on a small support set of labeled nodes from novel classes.
• The framework operates in a transductive setting, leveraging the full graph structure during inference to improve prediction accuracy.
• Two variants are proposed: TLP with self-supervised GCL (e.g., GraphCL, SUGRL) and TLP with supervised GCL (e.g., GraphCL*).
• The method bypasses episodic training and meta-optimization, instead relying on the generalization capacity of GCL-learned representations.
• Node representations are fine-tuned using a linear head with cross-entropy loss on the support set, while the GNN encoder remains frozen.
• Experiments compare TLP against meta-learning baselines under identical evaluation protocols across six real-world datasets.

Experimental results

Research questions

• RQ1 Can transductive linear probing with self-supervised GCL pretraining outperform meta-learning in few-shot node classification without any labeled data during pretraining?
• RQ2 How does the presence or absence of labels during GCL pretraining affect the performance of TLP in few-shot node classification?
• RQ3 Why do meta-learning methods underperform on datasets with fewer classes, such as CiteSeer, and how does TLP mitigate this?
• RQ4 To what extent does the quality of node representations learned via GCL determine the success of few-shot classification?
• RQ5 Is there a trade-off between scalability and adaptability between meta-learning and TLP frameworks in low-label regimes?

Key findings

• TLP with self-supervised GCL pretraining outperforms state-of-the-art meta-learning methods on both CoraFull and CiteSeer, even when no ground-truth labels are used during pretraining.
• On CiteSeer, the performance gap between meta-learning and TLP is larger than on CoraFull, indicating that meta-learning struggles more with fewer classes due to limited label diversity.
• TLP with self-supervised GCL achieves better clustering performance than TLP with supervised GCL, suggesting that avoiding label bias improves generalization to novel classes.
• t-SNE visualizations show that TLP learns more discriminative and separable node representations than meta-learning, especially on CiteSeer.
• The NMI and ARI scores of TLP are consistently higher than meta-learning across datasets, with the largest improvement observed on CiteSeer.
• TLP demonstrates superior scalability and generalization in low-supervision settings, highlighting its potential as a practical alternative to meta-learning.