[Paper Review] Graph Condensation for Graph Neural Networks

Wei Jin, Lingxiao Zhao+4 more|arXiv (Cornell University)|Oct 14, 2021

Advanced Graph Neural NetworksComputer Science47 references20 citations

TL;DR

This paper proposes Graph Condensation (GCond), a novel framework that distills large-scale attributed graphs into small, synthetic graphs while preserving GNN performance. By optimizing a gradient matching loss and parameterizing the graph structure as a function of learnable node features, GCond reduces graph size by over 99.9% and achieves up to 99.8% of the original test accuracy on Flickr and 95.3% on Reddit, enabling efficient training of diverse GNN architectures.

ABSTRACT

Given the prevalence of large-scale graphs in real-world applications, the storage and time for training neural models have raised increasing concerns. To alleviate the concerns, we propose and study the problem of graph condensation for graph neural networks (GNNs). Specifically, we aim to condense the large, original graph into a small, synthetic and highly-informative graph, such that GNNs trained on the small graph and large graph have comparable performance. We approach the condensation problem by imitating the GNN training trajectory on the original graph through the optimization of a gradient matching loss and design a strategy to condense node futures and structural information simultaneously. Extensive experiments have demonstrated the effectiveness of the proposed framework in condensing different graph datasets into informative smaller graphs. In particular, we are able to approximate the original test accuracy by 95.3% on Reddit, 99.8% on Flickr and 99.0% on Citeseer, while reducing their graph size by more than 99.9%, and the condensed graphs can be used to train various GNN architectures.Code is released at https://github.com/ChandlerBang/GCond.

Motivation & Objective

• To address the growing computational and storage burden of training GNNs on large-scale graphs with millions of nodes and edges.
• To explore whether a small, synthetic graph can be learned to train GNNs with performance comparable to training on the full original graph.
• To develop a method that simultaneously condenses both graph structure and node attributes in a way that preserves downstream GNN performance.
• To enable efficient training, storage, and retrieval of graph data by creating highly informative, minimal synthetic graphs.
• To demonstrate generalization of condensed graphs across different GNN architectures and downstream tasks like neural architecture search.

Proposed method

• The framework uses a gradient matching loss to align the training trajectory of GNNs trained on the condensed graph with those trained on the original graph.
• Graph structure in the condensed graph is parameterized as a differentiable function of the learnable node features, enabling end-to-end optimization.
• Node features in the condensed graph are treated as free parameters to be optimized during training.
• The method jointly optimizes the synthetic node features, graph structure (via attention-based edge weights), and labels to minimize performance gap.
• The approach is trained via backpropagation, with the loss measuring the L2 distance between gradients of GNN parameters on the original and condensed graphs.
• The framework is evaluated on multiple benchmark datasets including Reddit, Flickr, Citeseer, Cora, and Ogbg-arxiv, using various GNN architectures.

Experimental results

Research questions

• RQ1 Can a small synthetic graph be learned to train GNNs with performance comparable to training on the full original graph?
• RQ2 How can both graph structure and node attributes be effectively condensed while preserving GNN performance?
• RQ3 To what extent can the condensed graph generalize across different GNN architectures and hyperparameters?
• RQ4 How does the performance of GNNs trained on condensed graphs compare to those trained on the full dataset in neural architecture search (NAS) settings?
• RQ5 What structural and statistical properties do the condensed graphs exhibit, and how do they differ from the original graphs?

Key findings

• GCond reduces graph size by over 99.9% while achieving 95.3% of the original test accuracy on Reddit, 99.8% on Flickr, and 99.0% on Citeseer.
• The condensed graphs are significantly denser than the original graphs, with sparsity dropping from 0.09% to 2.57% on Reddit and 0.01% to 3.25% on Ogbg-arxiv.
• Homophily patterns are well-preserved in condensed graphs for Cora, Citeseer, and Flickr, though reduced in Reddit and Ogbg-arxiv.
• The condensed graph for Reddit exhibits a star-like structure, indicating that node features play a dominant role in information propagation.
• GNNs trained on condensed graphs generalize well across different architectures, including GCN, GraphSAGE, and SGC, with minimal performance drop.
• In neural architecture search (NAS), there is a strong correlation between performance on the condensed dataset and the full dataset, validating the utility of the condensed graphs.