[Paper Review] Learning by Sampling and Compressing: Efficient Graph Representation Learning with Extremely Limited Annotations

Xiaoming Liu, Qirui Li+4 more|arXiv (Cornell University)|Mar 13, 2020

Advanced Graph Neural Networks30 references4 citations

TL;DR

This paper proposes a novel training framework that combines adaptive sampling via multiple dependent random walks and tensor-train-based model compression to enable efficient graph representation learning with extremely limited labeled data. By reducing training data by up to 90% and compressing model parameters by over 6×, the method maintains or improves performance on GCN-based models across six baselines and three real-world datasets.

ABSTRACT

Graph convolution network (GCN) attracts intensive research interest with broad applications. While existing work mainly focused on designing novel GCN architectures for better performance, few of them studied a practical yet challenging problem: How to learn GCNs from data with extremely limited annotation? In this paper, we propose a new learning method by sampling strategy and model compression to overcome this challenge. Our approach has multifold advantages: 1) the adaptive sampling strategy largely suppresses the GCN training deviation over uniform sampling; 2) compressed GCN-based methods with a smaller scale of parameters need fewer labeled data to train; 3) the smaller scale of training data is beneficial to reduce the human resource cost to label them. We choose six popular GCN baselines and conduct extensive experiments on three real-world datasets. The results show that by applying our method, all GCN baselines cut down the annotation requirement by as much as 90$\%$ and compress the scale of parameters more than 6$ imes$ without sacrificing their strong performance. It verifies that the training method could extend the existing semi-supervised GCN-based methods to the scenarios with the extremely small scale of labeled data.

Motivation & Objective

• Address the challenge of training graph convolutional networks (GCNs) when labeled data is extremely scarce.
• Reduce human annotation cost by minimizing the required number of labeled nodes for training.
• Improve model generalization and reduce training deviation caused by poor sampling strategies.
• Enable existing GCN-based models to perform effectively under low-data regimes without architectural changes.
• Develop a generalizable framework that integrates sampling and compression for scalable, low-resource graph learning.

Proposed method

• Employ multiple dependent random walks to sample a representative subset of nodes, improving distributional coverage compared to uniform sampling.
• Use label density estimation to theoretically justify that random walk sampling yields more representative nodes than uniform sampling.
• Apply tensor-train (TT) decomposition to compress GCN weight matrices, reducing parameter count from $O(b imes c)$ to $O(d imes r imes \max(b,c))$ with $r \ll \min(b,c)$.
• Integrate the sampling and compression steps into a unified training pipeline: sample nodes first, then train the compressed model on the reduced dataset.
• Use dynamic programming to compute gradients efficiently during backpropagation on the TT-compressed model via chain rule on kernel matrices $G_k$.
• Train the compressed model using standard optimization with early stopping, using only the sampled nodes for supervision.

Experimental results

Research questions

• RQ1 Can a sampling strategy based on multiple dependent random walks outperform uniform sampling in terms of model performance when labeled data is extremely limited?
• RQ2 To what extent can tensor-train decomposition reduce the number of trainable parameters in GCN-based models without sacrificing performance?
• RQ3 Can the combination of adaptive sampling and model compression enable GCN-based models to achieve the same performance as full-data training with only 10–50% of the labeled data?
• RQ4 How does the proposed framework generalize across different GCN architectures and real-world graph datasets?
• RQ5 What is the trade-off between model compression and training time, and is the additional cost justified by performance and data efficiency gains?

Key findings

• All six GCN-based baselines achieved comparable or better performance when trained on just 10–50% of the original labeled data using the proposed framework.
• The method reduced the required annotation budget by up to 90% across all evaluated datasets and models.
• Model parameters were compressed by more than 6× using tensor-train decomposition, significantly reducing model size and training data dependency.
• The framework introduced only a 16% increase in training time, which is a reasonable trade-off for the gains in data efficiency and model compactness.
• Theoretical analysis confirmed that random walk sampling yields better label distribution coverage than uniform sampling, reducing training deviation.
• Empirical results on multi-label node classification tasks demonstrated consistent performance gains and robustness across diverse real-world graphs.