[Paper Review] Does Graph Distillation See Like Vision Dataset Counterpart?

Beining Yang, Kai Wang+6 more|arXiv (Cornell University)|Oct 13, 2023

Advanced Graph Neural Networks4 citations

TL;DR

This paper proposes Structure-broadcasting Graph Dataset Distillation (SGDD), a novel graph condensation method that explicitly preserves original graph structure by broadcasting adjacency matrices during synthetic graph generation. By minimizing Laplacian Energy Distribution (LED) shifts through spectral analysis, SGDD achieves state-of-the-art performance across 9 datasets, maintaining 98.6% of original YelpChi test accuracy with 1,000x smaller graph size and reducing LED shifts by 17.6%–31.4%.

ABSTRACT

Training on large-scale graphs has achieved remarkable results in graph representation learning, but its cost and storage have attracted increasing concerns. Existing graph condensation methods primarily focus on optimizing the feature matrices of condensed graphs while overlooking the impact of the structure information from the original graphs. To investigate the impact of the structure information, we conduct analysis from the spectral domain and empirically identify substantial Laplacian Energy Distribution (LED) shifts in previous works. Such shifts lead to poor performance in cross-architecture generalization and specific tasks, including anomaly detection and link prediction. In this paper, we propose a novel Structure-broadcasting Graph Dataset Distillation (SGDD) scheme for broadcasting the original structure information to the generation of the synthetic one, which explicitly prevents overlooking the original structure information. Theoretically, the synthetic graphs by SGDD are expected to have smaller LED shifts than previous works, leading to superior performance in both cross-architecture settings and specific tasks. We validate the proposed SGDD across 9 datasets and achieve state-of-the-art results on all of them: for example, on the YelpChi dataset, our approach maintains 98.6% test accuracy of training on the original graph dataset with 1,000 times saving on the scale of the graph. Moreover, we empirically evaluate there exist 17.6% ~ 31.4% reductions in LED shift crossing 9 datasets. Extensive experiments and analysis verify the effectiveness and necessity of the proposed designs. The code is available in the GitHub repository: https://github.com/RingBDStack/SGDD.

Motivation & Objective

• To address the limitation of existing graph distillation methods that overlook original structure information during synthetic graph generation.
• To investigate the impact of spectral structure shifts—specifically Laplacian Energy Distribution (LED) shifts—on cross-architecture generalization and downstream task performance.
• To develop a method that explicitly broadcasts original graph structure to synthetic graphs to reduce LED shifts and improve generalization.
• To validate the necessity and effectiveness of structure preservation in graph dataset distillation through empirical and theoretical analysis.
• To achieve state-of-the-art performance in both generalization and specific tasks like anomaly detection and link prediction using minimal graph size.

Proposed method

• Proposes SGDD, a structure-broadcasting framework that directly transfers the original adjacency matrix A to guide the generation of synthetic adjacency matrix A′, rather than inferring A′ from features.
• Introduces a spectral-based objective to minimize Laplacian Energy Distribution (LED) shifts between original and condensed graphs, ensuring structural fidelity.
• Employs a gradient matching strategy on features while explicitly preserving structural topology via direct broadcasting of A, decoupling structure learning from feature-based reconstruction.
• Uses a regularized objective with a coefficient β to control sparsity of the condensed graph, ensuring robustness and scalability.
• Applies optimal transport-based sampling during condensation to reduce computational cost while maintaining performance.
• Employs spectral analysis and shift coefficient (SC) as a proxy for LED shift to quantitatively evaluate structural preservation.

Experimental results

Research questions

• RQ1 To what extent do existing graph distillation methods fail to preserve original graph structure, and how does this affect performance?
• RQ2 What is the relationship between Laplacian Energy Distribution (LED) shifts and cross-architecture generalization performance in graph distillation?
• RQ3 Can explicit broadcasting of original graph structure improve performance on downstream tasks like link prediction and anomaly detection?
• RQ4 How does SGDD compare to vision-inspired graph distillation baselines (e.g., GCond) in terms of structural fidelity and generalization?
• RQ5 What is the sensitivity of SGDD to hyperparameters like β, and how does sampling size affect its scalability and performance?

Key findings

• SGDD reduces Laplacian Energy Distribution (LED) shifts by 17.6% to 31.4% across 9 datasets, demonstrating significant improvement in structural preservation.
• On the YelpChi dataset, SGDD maintains 98.6% of the original graph's test accuracy while reducing graph size by 1,000x, achieving state-of-the-art performance.
• SGDD improves cross-architecture generalization by an average of 9.6% over GCond across 5 GNN architectures (APPNP, Cheby, GCN, SAGE, SGC) on YelpChi.
• The method is robust to hyperparameter β, with performance stable across different sparsity levels, and achieves optimal results at 2,000 sampled nodes.
• Visualizations and ablation studies confirm that SGDD preserves homophily patterns and structural properties better than feature-only reconstruction methods.
• The condensed graphs in SGDD are significantly smaller (e.g., 153 nodes vs. 153k in Reddit) while maintaining comparable accuracy, with storage reduced from hundreds of MB to under 1 MB.